Interview Questions

An AI endpoint's median latency is stable, but p99 latency and cost per successful task are rising. How would you investigate?

How would you define service-level objectives and release policies for an AI assistant whose semantic correctness is not immediately observable?

Separate operational SLOs from sampled quality indicators and enforce critical safety constraints independently.

Design a canary release for a new model and prompt combination serving customer-facing traffic.

Use stable randomized assignment, comparable cohorts, hard safety stops, and predefined exposure and decision rules.

Implement deterministic canary assignment that remains stable across application instances and supports gradual expansion.

Hash a stable assignment key with an experiment salt and compare the resulting bucket with the rollout percentage.

How would you test fallback behavior when a primary AI provider becomes unavailable or returns unusable output?

Define failure classes and fallback eligibility explicitly, then inject faults and verify deadlines, contracts, safety parity, and user-visible behavior.

What are the limitations of shadow testing an AI agent that can invoke tools, and how would you make shadow execution safe?

Mirror only approved data and replace side-effecting tools with isolated simulations or read-only adapters.

A provider silently updates a hosted model alias and your tool-call failure rate doubles. What is your incident response?

Contain exposure, preserve evidence, compare artifact and provider metadata, and restore a qualified path while isolating the cause.

Design a test architecture for a product combining deterministic business rules, retrieval, LLM generation, and external tools.

Test each contract independently, then use representative end-to-end evaluations to measure composition failures.

How would you build and govern an evaluation dataset that remains representative without leaking into prompt tuning?

Separate development and held-out sets, split by meaningful groups or time, and manage provenance, access, and dataset refreshes.

How would you define ownership and release decision rights across SDET, ML engineering, application engineering, security, and product teams?

Assign accountable owners for system risks and evidence, with explicit promotion authority and escalation paths.

How would you evaluate competing AI vendors for a production testing or customer-support platform?

Run a workload-specific bake-off covering quality, reliability, security, operations, and total cost under contractual constraints.

What governance controls would you require before releasing an AI feature that processes sensitive customer data?

Use risk classification, approved data flows, enforceable controls, release evidence, and ongoing accountability.

How would you measure the ROI of introducing AI-assisted test generation into an SDET organization?

Measure net business value against a credible baseline, including review effort, maintenance, infrastructure, and escaped-defect risk.

You inherit an SDET team with strong automation skills but little AI experience. What is your first 90-day strategy?

Baseline risks and capabilities, deliver one bounded pilot, and build reusable evaluation and operational practices around it.

Product wants to release an AI feature despite inconclusive quality results. How would you lead the decision?

Make the uncertainty and consequences explicit, then offer risk-bounded options rather than presenting an unsupported pass or fail.

How would you decide which AI outputs need human review and verify that the review process actually reduces risk?

Route by consequence and validated risk signals, then measure reviewer performance, workload, and missed failures.

Design a shared AI evaluation platform for multiple product teams without forcing every team into the same quality metric.

Standardize execution, provenance, security, and reporting while allowing domain-specific datasets, evaluators, and acceptance policies.

Design the quality and release architecture for a multi-tenant enterprise RAG assistant with strict document permissions.

Enforce identity-aware retrieval and evidence access, then evaluate retrieval, grounding, isolation, freshness, and end-to-end outcomes separately.

Design a safe autonomous agent that can propose and execute customer refunds, including its testing strategy.

Keep financial authority in deterministic policy services and treat the model's tool calls as untrusted proposals.

Implement asynchronous primary-to-fallback routing with a shared deadline and restricted fallback eligibility.

Give the primary a bounded share of the deadline, use only the remaining time for fallback, and propagate non-retryable failures.

Design an evaluation strategy for a multimodal system that extracts invoice data from images and initiates an approval workflow.

Evaluate perception, structured extraction, business validation, and workflow behavior as separate layers with end-to-end risk checks.

Design the test and observability strategy for a real-time voice assistant with interruption support and strict response-latency targets.

Measure the streaming pipeline end to end and test conversational state under partial transcripts, cancellations, and network faults.

Design a multi-tenant AI gateway that enforces spend limits while maintaining fairness, latency, and data isolation.

Combine authenticated tenant routing, atomic budget reservations, bounded concurrency, approved model policies, and usage reconciliation.

How would you secure and test the supply chain of an AI system that loads third-party models, evaluation plugins, and retrieval data?

Treat models, executable plugins, datasets, and indexes as separate supply-chain trust boundaries with verified provenance and constrained execution.

Implement reciprocal rank fusion for hybrid retrieval and describe how you would evaluate its effect on a RAG system.

Fuse ranked lists using reciprocal rank contributions, then test both retrieval relevance and downstream grounded-answer quality.

Design a reproducible evaluation harness for an agent whose behavior depends on time, external APIs, and mutable business state.

Control deterministic dependencies through versioned fixtures and stateful simulation, while measuring model variance separately.

A model rollout improves average task success but increases spend, causes occasional cross-tenant citations, and overloads human reviewers. How would you lead diagnosis and redesign?

Contain the isolation failure immediately, stabilize review and cost controls, then investigate each failure path before considering renewed rollout.

What is risk-based testing, and why does it matter in Manual Testing?

risk-based testing is a core Manual Testing concept used to make testing decisions more accurate, repeatable, and aligned with product risk.

How would you apply risk-based testing in a production Manual Testing project?

Apply risk-based testing through an explicit, repeatable design: define the expected behavior, automate the highest-risk path, capture useful diagnostics, and verify the result against a stable oracle.

What is equivalence partitioning, and why does it matter in Manual Testing?

equivalence partitioning is a core Manual Testing concept used to make testing decisions more accurate, repeatable, and aligned with product risk.

How would you apply equivalence partitioning in a production Manual Testing project?

Apply equivalence partitioning through an explicit, repeatable design: define the expected behavior, automate the highest-risk path, capture useful diagnostics, and verify the result against a stable oracle.

What is boundary value analysis, and why does it matter in Manual Testing?

boundary value analysis is a core Manual Testing concept used to make testing decisions more accurate, repeatable, and aligned with product risk.

How would you apply boundary value analysis in a production Manual Testing project?

Apply boundary value analysis through an explicit, repeatable design: define the expected behavior, automate the highest-risk path, capture useful diagnostics, and verify the result against a stable oracle.

What is decision-table testing, and why does it matter in Manual Testing?

decision-table testing is a core Manual Testing concept used to make testing decisions more accurate, repeatable, and aligned with product risk.

How would you apply decision-table testing in a production Manual Testing project?

Apply decision-table testing through an explicit, repeatable design: define the expected behavior, automate the highest-risk path, capture useful diagnostics, and verify the result against a stable oracle.

What is state-transition testing, and why does it matter in Manual Testing?

state-transition testing is a core Manual Testing concept used to make testing decisions more accurate, repeatable, and aligned with product risk.

How would you apply state-transition testing in a production Manual Testing project?

Apply state-transition testing through an explicit, repeatable design: define the expected behavior, automate the highest-risk path, capture useful diagnostics, and verify the result against a stable oracle.

What is pairwise testing, and why does it matter in Manual Testing?

pairwise testing is a core Manual Testing concept used to make testing decisions more accurate, repeatable, and aligned with product risk.

How would you apply pairwise testing in a production Manual Testing project?

Apply pairwise testing through an explicit, repeatable design: define the expected behavior, automate the highest-risk path, capture useful diagnostics, and verify the result against a stable oracle.

What is exploratory testing charters, and why does it matter in Manual Testing?

exploratory testing charters is a core Manual Testing concept used to make testing decisions more accurate, repeatable, and aligned with product risk.