Search topics, questions, scenarios…

AI quality engineering

AI Testing Lab

Work through realistic AI-system failures before opening the evidence-based solution. Progress follows your account when signed in.

Practise in an interview
0%0/12 tasks

Lab completion

Mark a task complete only after you can explain the test oracle, evidence, and release decision.

Prompt regressionMedium

Build a versioned prompt regression gate

A support-answer prompt changes weekly. Build a gate that catches factual, refusal, and format regressions without exact-string assertions.

Setup

You have 40 reviewed cases with required facts, forbidden claims, and a JSON response contract.

  1. 1Define case-level rubrics and hard contracts
  2. 2Run each case repeatedly against the candidate prompt
  3. 3Aggregate pass rates with confidence bounds
  4. 4Block only on agreed safety and quality thresholds

Keep schema checks deterministic

Version the prompt, model, dataset, and evaluator together

RAG evaluationHard

Isolate retrieval from generation failures

A policy assistant gives a wrong answer. Design diagnostics that identify whether retrieval, context assembly, or generation failed.

Setup

The system exposes retrieved document IDs, ranked chunks, assembled context, citations, and final response.

  1. 1Create evidence-labeled queries
  2. 2Measure retrieval recall and ranking separately
  3. 3Verify context retention and access filters
  4. 4Evaluate groundedness and answer completeness

A good answer with the wrong evidence is still a failure

Include unanswerable and permission-restricted cases

Contracts and toolsMedium

Create a contract-faithful LLM test double

Unit tests are slow and flaky because every orchestration test calls a hosted model. Replace it without pretending mocks prove model quality.

Setup

The application supports structured outputs, tool calls, refusals, rate limits, and streamed text.

  1. 1Model the provider boundary
  2. 2Create fixtures for every response class
  3. 3Simulate malformed and partial streams
  4. 4Keep separate live-model evaluations

Mock protocol behavior, not intelligence

Include ordering and duplicate-event cases

Safety and privacyHard

Red-team indirect prompt injection in RAG

An indexed document contains instructions telling the assistant to reveal secrets and ignore access controls.

Setup

The assistant retrieves internal documents and can call read-only account tools.

  1. 1Create direct and indirect injection payloads
  2. 2Vary payload placement and encoding
  3. 3Assert authorization outside the model
  4. 4Verify safe refusal and audit evidence

Retrieved text is untrusted data

A prompt instruction is not an authorization boundary

Safety and privacyMedium

Detect PII leakage across prompts, traces, and outputs

A test-data generator uses support tickets. Prove that personal data is not sent, logged, or reproduced.

Setup

Synthetic tickets include names, emails, phone numbers, account IDs, and canary secrets.

  1. 1Map every data sink
  2. 2Inject unique canaries
  3. 3Inspect outbound requests and telemetry
  4. 4Test output memorization and deletion paths

Redaction after logging is too late

Regex alone misses contextual identifiers

Contracts and toolsMedium

Test structured output and tool-call contracts

An agent creates invalid tool arguments and occasionally repeats a payment-side effect after a retry.

Setup

The orchestration layer validates JSON Schema and requires approval for mutating tools.

  1. 1Generate valid and invalid argument boundaries
  2. 2Test unknown and missing fields
  3. 3Simulate retries and duplicate call IDs
  4. 4Assert approval and idempotency rules

Validation must happen before execution

The model is not trusted to guarantee exactly-once behavior

OperationsHard

Enforce quality, latency, and cost budgets

A release improves answer quality but doubles p95 latency and triples token cost. Build a balanced release gate.

Setup

You have per-request token usage, model latency, end-to-end latency, cache status, and quality scores.

  1. 1Define service-level indicators
  2. 2Segment by request class
  3. 3Compare baseline and candidate
  4. 4Set hard safety gates and negotiated performance budgets

Averages hide tail latency

Cache hits and misses need separate reporting

OperationsHard

Detect model, data, and behavior drift

Quality drops slowly after knowledge updates and provider model changes. Design monitoring that detects drift without logging sensitive prompts.

Setup

The platform records version IDs, privacy-safe features, evaluation outcomes, user corrections, latency, and costs.

  1. 1Separate drift types
  2. 2Create stable canary evaluations
  3. 3Monitor production slices
  4. 4Define alert and rollback policy

A provider alias can change behavior

Aggregate quality can hide one damaged customer segment

AI-assisted automationMedium

Cluster automation failures without hiding root causes

A regression run produces 600 failures. Use AI-assisted triage while preserving evidence and human accountability.

Setup

Artifacts include stack traces, normalized errors, test metadata, screenshots, DOM snapshots, and deployment changes.

  1. 1Normalize deterministic signatures
  2. 2Cluster with explainable evidence
  3. 3Summarize each cluster
  4. 4Measure assignment precision and analyst correction

Start with deterministic grouping

Never let a summary replace source artifacts

AI-assisted automationEasy

Review AI-generated tests before merge

A coding assistant produces 30 passing UI tests. Build a review checklist that prevents false confidence and maintenance debt.

Setup

Generated tests compile and pass against the current environment but were not reviewed for intent or fault detection.

  1. 1Trace every test to a risk or requirement
  2. 2Inspect assertions and negative paths
  3. 3Mutation-check important tests
  4. 4Review isolation, data, waits, locators, and ownership

Passing is not evidence the test can fail correctly

Count unique risks, not generated test methods

Prompt regressionHard

Design metamorphic tests for variable answers

There is no single expected wording for a summarizer. Define transformations whose relationships should still hold.

Setup

The summarizer must preserve facts, ignore formatting noise, respect requested language, and omit irrelevant boilerplate.

  1. 1Choose valid input transformations
  2. 2Define expected output relations
  3. 3Run repeated trials
  4. 4Investigate relation violations with source evidence

Do not assume every paraphrase is semantically identical

Some transformations legitimately change emphasis

OperationsHard

Design an enterprise AI quality architecture

Multiple teams are shipping assistants with inconsistent prompts, evaluations, traces, and release criteria.

Setup

Products differ in risk, data sensitivity, model provider, retrieval stack, and deployment cadence.

  1. 1Define shared contracts and evidence
  2. 2Provide reusable evaluation infrastructure
  3. 3Allow risk-tier policy profiles
  4. 4Create ownership and incident workflows

Central standards should not force one prompt framework

High-risk actions need stronger controls than summarization