Codemify · AI Testing Mastery

Master

AI Testing

Before AI Tests You.

Recorded lessons, live Q&A sessions, and internship - everything you need to master AI testing at your own pace, with expert support every week.

Recorded lessons

Live Q&A sessions

Hands-on exercises

Internship from Day 1

Cohort Dates

July 11 - Aug 22

Duration

6 Weeks

Timezone

California (PT)

Live Q&A Sessions

Mon & Thu · 7:30 PM CA · 1 hr

Saturday · 9:00 AM CA · 1 hr

Enroll Now View Full Curriculum

Who this is for

Built for QA Engineers
Ready for the AI Era

AI features are shipping every sprint. Your testing skills were not built for this. This course changes that - in 6 live weeks.

This is for you if you are a...

QA engineer or SDET who wants to stay ahead of the AI wave instead of being replaced by it
Manual or automation tester ready to move into the fastest-growing specialization in the industry
Junior to mid-level QA professional who wants to land their first AI-focused role
Senior QA engineer who needs to lead AI quality strategy and needs the right frameworks

The Outcome Is Simple

In 6 weeks, you'll be one of the very few QA engineers who can walk into an interview, open a laptop, and demonstrate exactly how to evaluate an LLM, run a red team attack, and integrate AI quality gates into a CI/CD pipeline. That's not a soft skill. That's a career-defining advantage.

Enroll Now!

Download the Program

Course Program

Week 1 - Foundation:

Topics:

How LLMs work from a tester's point of view: tokens, temperature, non-determinism, context window
Why traditional testing doesn't work on AI outputs
Where does AI testing fit into the testing pyramid
Categories of AI failures: hallucination, bias, accuracy, latency, cost blowouts
What does "quality" mean when outputs vary between runs

Cross-model evaluation of commercial LLMs (OpenAI, Anthropic)
Core failure-mode probing: hallucination, factual accuracy, and instruction-following
Non-determinism and temperature analysis, with manual output evaluation using property-based checks and scoring rubrics

Week 2 - Prompt Evaluation with Promptfoo

Topics:

System Prompt vs User Prompt
Promptfoo architecture (promptfoo yaml, providers, tests, assertions),
Assertion types (deterministic vs non-deterministic),
Running evals across multiple prompts × models × providers- Viewing results on promptfoo web viewer and analyzing the metrics

Provision API keys (OpenAI, Anthropic)
Running evaluations with different prompts, assertions and providers and comparing the results
Expand and write your own deterministic tests
Implement LLM as a judge assertions to verify non-deterministic outputs

Week 3 - Testing Quality: Cost, Model Selection & RAG

Topics:

Importance of evaluating the cost of different models
Evaluating models on quality dimensions: factuality, accuracy, relevance
When to use embedding similarity vs LLM-as-judge vs deterministic assertions
Evaluating hosted open-weight models with Promptfoo
Testing RAG: retrieval quality and faithfulness, the two-layer failure model

Cost-evaluation exercise across commercial and open-weight models
Connect a hosted open-weight model (Groq / OpenRouter, OpenAI-compatible) to Promptfoo
Run automated evals across commercial and open-weight models
Reverse-engineer an effective system prompt from an AI application
RAG, and single/multi-turn conversation testing

Week 4 - Weighted assertions, debugging, CSV file assertions, red teaming intro

Topics:

Using weighted assertions in promptfoo evals
Different AI testing tools (Deep Eval, Lang Chain)
Real example of AI failure in production and how to create evaluations that would prevent that
Introduction to AI red teaming
Differences between traditional cyber security testing and AI red teaming

Debugging Exercise - resolve problems in the YAML files
CSV Exercise - Practice with the CSV file assertions
How to share evaluation results with your team
Manually trying to break an AI system through adversarial attacks

Week 5 - Red Teaming Deep Dive

Topics:

Promptfoo red teaming configuration
White box vs black box testing
Defining the threat model and attack strategy
Types of LLM Vulnerabilities
OWASP LLM Top 10 and different frameworks

Exercise on how to define a red teaming strategy on a black box vs a white box scenario
Creating and running first red teaming tests and analyzing the results
Experiment with multiple plugins and strategies

Week 6 - Red Teaming Advanced

Topics

How red teaming would fit into the SDLC
How to refine the red teaming vulnerability results
How to avoid false positives using grader guidance and examples
Guardrails for AI vulnerabilities remediation
CI/CD AI Test integration Example

Implement grader guidance to prevent false positives
Write targeted regression tests so the same exploits can't reappear

Join Now!

Our YouTube Channel

Learn from the Experts

250+

videos

30k+

subscribers

2+ mln

views

We believe in sharing knowledge, so on our channel, you'll find:

Student success stories

Tutorials and useful materials for aspiring QA engineers

Tips on interviews and job opportunities in the QA field

With over 1 million views, our channel has become a trusted resource for learning and growth in the QA industry.

Subscribe

Student Success Stories

Nothing speaks louder than real success stories. Our students come from different backgrounds, but they all have one thing in common — they successfully transitioned into QA careers after completing our program.