Season 8: You Own the Release · Episode 5 of 8
The Tests Passed. The AI Got Worse.
You play: AI Engineering Intern
Included with Core programme and Certification prep
The situation
A new prompt for assist-service is friendlier, cheaper to write and passes every unit and integration test. Amara has scored its answers against their source documents, and they are measurably worse.
What you'll learn
- Separate unit, integration, security and AI-evaluation gates; AI-quality metrics with sample size and margin of error; conservative thresholds; holdout contamination; releasing gradually with a canary; rolling back to the last known-good release; and recording a release decision that says what the measurement can and can't support.
- Blue-green, canary, ring-based and A/B strategies, feature flags with Azure App Configuration, deployment slots and swap, the hotfix path and backward-compatible database changes.
Who you work with
Amara Okafor
ML and Data Engineer
Maya Chen
Senior AI Engineer · Your mentor
Kabelo Mokoena
Platform and DevOps Engineer
Scenes
- 1.Every test is green
- 2.A code test, or a quality check?
- 3.Four separate gates
- 4.It's already live
- 5.A fix that passes, and a fix that's honest
- 6.0.90, on forty questions
- 7.Handover
What you leave with
You separated the pipeline into unit, integration, security and AI-evaluation gates and proved they let a good change through and stopped a bad one. You rolled back a live release exactly one step, refused a fix whose score came from a holdout it was tuned on, shipped an honest fix as a canary with an independent approval, and recorded why a score on 40 questions can't support a release.