Codemify · AI Testing Mastery
Master
AI Testing
Before AI Tests You.
Recorded lessons, live Q&A sessions, and hands-on exercises - everything you need to master AI testing at your own pace, with expert support every week.
Recorded lessons
Live Q&A sessions
Hands-on exercises
Cohort Dates
June 13 - July 25
Duration
6 Weeks
Timezone
California (PT)
Live Q&A Sessions
Mon & Thu · 6:00 PM CA · 1 hr
Saturday · 9:00 AM CA · 1–1.5 hrs
Who this is for Built for QA Engineers
Built for QA Engineers
Ready for the AI Era
AI features are shipping every sprint. Your testing skills were not built for this. This course changes that - in 6 live weeks.
This is for you if you are a...- QA engineer or SDET who wants to stay ahead of the AI wave instead of being replaced by it
- Manual or automation tester ready to move into the fastest-growing specialization in the industry
- Junior to mid-level QA professional who wants to land their first AI-focused role
- Senior QA engineer who needs to lead AI quality strategy and needs the right frameworks
The Outcome Is Simple
In 6 weeks, you'll be one of the very few QA engineers who can walk into an interview, open a laptop, and demonstrate exactly how to evaluate an LLM, run a red team attack, and integrate AI quality gates into a CI/CD pipeline. That's not a soft skill. That's a career-defining advantage.
Course Program
Download Full Course Program Here
Just type your email
Week 1 - Foundation:
Topics:
- How LLMs work from a tester's point of view: tokens, temperature, non-determinism, context window
- Why traditional testing doesn't work on AI outputs
- Where does AI testing fit into the testing pyramid
- Categories of AI failures: hallucination, bias, accuracy, latency, cost blowouts
- What does "quality" mean when outputs vary between runs
- Set up a local OpenAI/Anthropic API keys
- Set up LM Studio for testing local LLMs
- Test different AI/LLM failure modes manually on OpenAI, Anthropic, and local models
- Observe non-determinism at different temperatures
Week 2 - Prompt Evaluation with Promptfoo
Topics:
- System Prompt vs User Prompt
- Promptfoo architecture (promptfoo yaml, providers, tests, assertions),
- Assertion types (deterministic vs non-deterministic),
- Running evals across multiple prompts × models × providers- Viewing results on promptfoo web viewer and analyzing the metrics
- Running evaluations with different prompts, assertions and providers and comparing the results
- Expand and create your own tests
- Implement LLM as a judge assertions to verify non-deterministic outputs
Week 3 - Testing Quality: Accuracy, Relevance, Factuality
Topics:
- Importance of evaluating costs of different models
- Evaluating different models based on other factors (like factuality, accuracy, relevance, etc)
- When to use embedding similarity vs LLM as judge vs deterministic assertions
- How to evaluate local LLM models using Promptfoo
- Exercise to evaluate costs of different models
- Connect LM Studio with promptfoo to run a local LLM
- Run evals against commercial and local models to compare the results
- Generate a system prompt by reverse engineering an AI application
Week 4 - Weighted assertions, debugging, CSV file assertions, red teaming intro
Topics:
- Using weighted assertions in promptfoo evals
- Different AI testing tools (Deep Eval, Lang Chain, Arato.ai, Agenta AI)
- Real example of AI failure in production and how to create evaluations that would prevent that
- Introduction to AI red teaming
- Differences between traditional cyber security testing and AI red teaming
- Writing evals using [arato.ai](http://arato.ai)
- Debugging Exercise - resolve problems in the YAML files
- CSV Exercise - Practice with the CSV file assertions
- How to share evaluation results with your team
- Manually trying to break an AI system through adversarial attacks
Week 5 - Red Teaming Deep Dive
Topics:
- Promptfoo red teaming configuration
- White box vs black box testing
- Defining the threat model and attack strategy
- Types of LLM Vulnerabilities
- OWASP LLM Top 10 and different frameworks
- Exercise on how to define a red teaming strategy on a black box vs a white box scenario
- Creating and running first red teaming tests and analyzing the results
- Experiment with multiple plugins and strategies
Week 6 - Red Teaming Advanced
Topics
- How red teaming would fit into the SDLC
- How to refine the red teaming vulnerability results
- How to avoid false positives using grader guidance and examples
- Guardrails for AI vulnerabilities remediation
- CI/CD AI Test integration Example
- Implement grader guidance to prevent false positives
- Write targeted regression tests so the same exploits can't reappear
🗓️ Month 1: Tabular Data Mastery & Foundations
- Focus: Mastering the data science toolkit, EDA, and core regression models.
- Project: House Prices - Advanced Regression Techniques
- You'll build models to predict home values, mastering feature engineering and regression on a complex, real-world Kaggle dataset.
- (Skills: EDA, Feature Engineering, Regression Models, XGBoost, Kaggle Competitions)
Main Project Goal: Build the v0.1 command-line version with simple, rule-based responses.
Our YouTube Channel
Learn from the Experts
250+
videos
30k+
subscribers
2+ mln
views
We believe in sharing knowledge, so on our channel, you'll find:
Student success stories
Tutorials and useful materials for aspiring QA engineers
Tips on interviews and job opportunities in the QA field
With over 1 million views, our channel has become a trusted resource for learning and growth in the QA industry.
Student Success Stories
Nothing speaks louder than real success stories. Our students come from different backgrounds, but they all have one thing in common — they successfully transitioned into QA careers after completing our program.
Reviews on Google
4.9
124 reviews
Our students work at
100% Money-Back Guarantee
If the course isn’t right for you, we offer a full refund within 14 days after the course starts.
© 2025 Codemify. All rights reserved.
