AI for SDET Interview Questions

How does testing an LLM-powered feature differ from testing a conventional rules-based feature?

What are tokens and context windows, and how can they cause failures in an AI-assisted test-generation workflow?

Tokens are model-specific text units, and context limits constrain how much input and output a request can contain.

How would you test an embedding-based search service used to retrieve similar defects?

Measure retrieval relevance on labeled queries and separately verify filtering, indexing, and access-control behavior.

Does setting temperature to zero make LLM tests deterministic?

No. It reduces sampling variability but does not guarantee identical outputs across requests or environments.

How would you detect and reduce hallucinated APIs in AI-generated automation code?

Validate generated code against the repository's actual dependencies, then compile, execute, and review it.

Implement recall at k for a RAG retriever and explain what this metric cannot tell you.

Recall at k measures how many labeled relevant documents appear in the first k results; it does not measure answer correctness.

When would you choose prompting, RAG, or fine-tuning for an internal test-generation assistant?

Use prompting for task instructions, RAG for current repository knowledge, and fine-tuning for demonstrated recurring behavior gaps.

How would you design a defensible evaluation of an LLM that classifies test failures by root cause?

Use held-out, adjudicated incidents, repeated runs, per-class metrics, and explicit handling of uncertainty.

What controls would you require before sending production failure logs to an external LLM?

Minimize and sanitize data, verify provider handling terms, enforce access controls, and audit the complete data path.

Why is a tool-calling test agent riskier than a text-only assistant, and where should controls live?

Tool calls can change systems, so authorization and validation must be enforced outside the model.

How would you qualify a provider model upgrade for an AI testing platform?

Run paired offline evaluations, verify integration contracts, then use a monitored canary with rollback capability.

What information should a prompt contain to generate maintainable automated tests from a user story?

Provide verified requirements, framework constraints, repository conventions, expected assertions, and a clear output contract.

How would you select few-shot examples for a prompt that converts defects into regression tests?

Choose correct, diverse examples that demonstrate the desired reasoning evidence and test conventions without leaking evaluation cases.

Implement a testable prompt builder that keeps trusted instructions separate from an untrusted defect report.

Build separate messages, serialize the report as data, enforce input limits, and test message construction deterministically.

How should a test-generation assistant handle conflicting requirements, examples, and repository conventions?

Use an explicit source-precedence policy and surface unresolved conflicts instead of silently choosing a convenient interpretation.

How would you prompt an LLM to discover useful edge cases without producing a long list of redundant tests?

Provide explicit boundaries and ask for risk-ranked equivalence classes with requirement references.

How can metamorphic testing evaluate a prompt when there is no single correct answer?

Define justified relationships between related inputs and check whether outputs preserve those relationships.

Validate structured LLM-generated test plans and distinguish schema validation from semantic validation.

Parse against a strict schema, then check domain rules such as unique identifiers and valid requirement references.

How would you test indirect prompt injection through repository files or retrieved documentation?

Place malicious instructions in untrusted sources and verify both model behavior and independently enforced action boundaries.

How would you version and regression-test prompts used in a shared SDET platform?

Treat prompts as versioned production artifacts and evaluate changes against fixed, representative cases before release.

An assistant ignores a requirement even though it was included in the request. How would you debug the prompt pipeline?

Inspect the actual assembled request, verify context integrity, and isolate competing instructions before rewriting the prompt.

What should you request from an LLM to make a failure-triage recommendation auditable?

Request a concise conclusion, supporting evidence, uncertainty, alternatives, and next verification steps.

Design a safe CI workflow for AI-generated test changes.

Generate a reviewable patch in an isolated environment, run layered checks, and require approval before merging.

An LLM generates a flaky Playwright cart test. Rewrite it to control backend data and assert user-visible behavior.

Use scoped network fixtures, accessible locators, and retrying assertions instead of arbitrary sleeps.

What would you review in AI-generated tests beyond whether they pass?

Review test intent, oracle independence, failure sensitivity, isolation, security, and maintainability.

How would you use AI to diagnose flaky tests without allowing it to hide real product defects?

Use AI to rank evidence-backed hypotheses, then verify them with controlled experiments before changing tests.

Implement bounded retries for an LLM API adapter and explain how to test them deterministically.

Retry only classified transient failures, cap attempts, and inject the operation and sleeper for deterministic tests.

Should an AI self-healing locator automatically replace a broken selector in production test suites?

It may propose a replacement, but semantic equivalence and test intent must be verified before accepting it.

Implement metrics for a judge that decides whether generated tests are acceptable, and explain the danger of false acceptance.

Compare judge decisions with adjudicated labels and track false acceptance separately from aggregate accuracy.

How would you validate AI-based test selection before allowing it to skip tests in CI?

Run it in shadow mode against full-suite outcomes and measure missed failures, not just runtime savings.

How would you use AI to generate API tests from an OpenAPI specification without overtrusting the specification?

Generate contract-focused cases from the schema, then supplement them with business invariants and independently reviewed security scenarios.

Write a deterministic UI test for an AI assistant's abstention response and explain where live-model evaluation belongs.

Mock the model-facing application API for the UI test and evaluate real abstention behavior in a separate suite.

How would you performance-test an LLM-backed failure-analysis service while controlling cost and avoiding misleading results?

Test infrastructure with controlled substitutes, then run bounded live load tests with representative token sizes and provider limits.

Design an AI agent that investigates a failed build and proposes a test fix without independently weakening quality controls.

Separate evidence collection, proposal generation, sandbox validation, and human approval behind explicit policy boundaries.

How would you lead a rollout of AI-assisted testing and demonstrate value without incentivizing low-quality test generation?

Start with a bounded pilot and measure verified engineering outcomes, review burden, and safety—not generated test counts.

How would you design a test strategy for a RAG assistant without treating every failure as a model-quality problem?

Test ingestion, retrieval, context assembly, generation, and application controls separately, then connect them with end-to-end scenarios.

What makes a golden evaluation dataset trustworthy and maintainable for an enterprise RAG system?

Use representative, versioned cases with evidence-backed expectations, explicit rubrics, and protected holdout data.

Implement retrieval metrics for ranked document IDs and explain what recall@k, precision@k, and reciprocal rank fail to measure.

Measure evidence coverage and ranking separately; retrieval metrics alone do not establish answer quality.

How would you detect ingestion and chunking regressions before they appear as incorrect RAG answers?

Verify source fidelity, metadata preservation, chunk boundaries, and evidence retrievability using controlled documents.

How would you evaluate groundedness when an answer combines supported facts with plausible but unsupported conclusions?

Decompose the answer into verifiable claims and assess whether each claim is supported by the supplied evidence.

How do answer correctness, groundedness, relevance, and completeness differ in an LLM evaluation?

They measure truth or task validity, evidence support, responsiveness, and coverage of required information, respectively.

Write a validator for an LLM-generated refund decision that enforces JSON structure and business rules.

Validate strict JSON, schema constraints, and cross-field business invariants before consuming the output.

How would you test citation quality beyond checking whether an answer contains links?

Check citation validity, claim support, coverage, provenance, and user authorization.

How would you evaluate whether a RAG assistant answers, abstains, or asks for clarification at the right time?

Label answerability and ambiguity explicitly, then measure both unsafe answers and unnecessary abstentions.

How do you make LLM regression tests useful when repeated executions can produce different answers?

Test behavioral invariants, repeat representative cases, and report distributions rather than requiring identical text.

Implement a metamorphic test for a RAG assistant when a complete reference answer is unavailable.

Apply transformations with known semantic effects and assert task-specific invariants rather than identical wording.

How would you test a multi-turn assistant's memory, corrections, and context-window behavior?

Evaluate conversation trajectories, including state updates, explicit corrections, topic changes, and context truncation.

How would you evaluate a multilingual RAG assistant without assuming translated English tests are sufficient?

Use native-language cases, locale-specific expectations, and cross-language retrieval tests with per-language reporting.

How would you validate an LLM-as-judge before using its scores in release decisions?

Calibrate it against independently labeled human examples and measure its errors, biases, and stability.

Implement a paired bootstrap confidence interval for the quality difference between two systems evaluated on the same cases.

Resample case-level paired differences and estimate uncertainty around the mean candidate-minus-baseline score.

Design release gates for an LLM application that balance quality, safety, latency, and evaluation cost.

Use staged gates with deterministic contracts, targeted behavioral suites, statistical quality checks, and non-negotiable safety boundaries.

Why can a higher aggregate evaluation score hide a serious regression, and how would you prevent that?

Report meaningful slices, stable weighting, sample sizes, and critical failure counts alongside the overall score.

How would you build a reliable human annotation workflow for ambiguous LLM answers?

Use explicit rubrics, independent overlapping reviews, evidence access, adjudication, and annotation-quality monitoring.

Write tests proving that a tool-execution boundary rejects unauthorized model-generated calls before any side effect occurs.

Treat tool calls as untrusted proposals and verify deny-before-execute behavior using server-side identity and permissions.

How would you test tool-calling behavior under timeouts, retries, partial failures, and malformed tool responses?

Inject failures at execution boundaries and verify bounded retries, idempotency, reconciliation, and truthful user-facing outcomes.

How would you evaluate an agent that can achieve a task through several different valid tool sequences?

Score verified outcomes and execution constraints rather than requiring one exact reference trajectory.

How would you performance-test a streaming RAG application while preserving visibility into answer quality?

Measure end-to-end and stage-level latency, streaming behavior, throughput, errors, and cost on a representative workload.

What observability would you add so an SDET can diagnose and replay an LLM failure without overcollecting sensitive data?

Capture versioned execution metadata and privacy-controlled artifacts linked by a trace ID.

How would you test resistance to direct prompt injection without confusing harmless user instructions with security violations?

Define protected behaviors and trust boundaries, then measure whether adversarial user input causes a prohibited outcome.

Implement an indirect prompt-injection regression test for instructions embedded in retrieved content.

Place a controlled malicious instruction in a relevant document and inspect both the answer and trusted execution trace.

How would you test tenant isolation across retrieval, caching, conversation memory, and tool execution?

Seed distinguishable tenant data and verify isolation at every stateful and authorization boundary.

How would you assess sensitive-data leakage without using real credentials or customer records in your tests?

Use synthetic secrets and privacy fixtures, then inspect every output and persistence channel.

How would you test an agent's resistance to data exfiltration and server-side request forgery through network tools?

Test enforced destination and payload policies using controlled endpoints, with the model treated as an untrusted caller.

How would you security-test model-generated JSON, HTML, Markdown, SQL, or shell-like content consumed by downstream systems?

Treat generated output as untrusted input and test the sink-specific controls that prevent execution or unsafe interpretation.

How would you evaluate bias in an AI assistant while avoiding simplistic demographic score comparisons?

Combine task-specific harm definitions, controlled counterfactuals, representative subgroup cases, and uncertainty-aware analysis.

How would you test resilience to poisoned documents and knowledge-base updates in a RAG system?

Inject controlled misleading content into a test corpus and evaluate ingestion trust, retrieval influence, provenance, and recovery.

Implement a synthetic-canary scanner for leakage in user responses, tool payloads, and logs, and explain its limits.

Generate a unique fake marker and scan serialized artifacts for exact and explicitly supported encoded forms.

How would you organize an authorized red-team program for an enterprise AI agent?

Start with threat models and rules of engagement, then turn verified findings into mitigations and regression tests.

How would you monitor production AI quality and detect drift without assuming offline evaluation predicts every failure?

Combine operational signals, privacy-aware sampled evaluation, user outcomes, and version-attributed change detection.

A candidate model improves the automated score but domain experts identify new serious errors. How would you lead the release decision?

Pause the decision, investigate disagreement at the case level, and apply predeclared risk criteria rather than choosing the most favorable metric.

How would you version an AI application's model, prompts, retrieval configuration, and evaluation data so a production result can be traced to a release?

Create an immutable release manifest that references every behavior-changing artifact and record its identifier with each execution.

How would you structure CI quality gates for an LLM feature without making every pull request slow and expensive?

Use risk-based test tiers: deterministic checks on every change, targeted evaluations on pull requests, and broader statistical evaluations before promotion.

Implement a paired statistical quality gate that rejects a candidate when non-inferiority to the baseline has not been demonstrated.

Compute paired score differences and require their one-sided lower confidence bound to exceed a predefined non-inferiority margin.

How would you distinguish AI evaluation flakiness from a genuine quality regression in CI?

Separate infrastructure failures, generation variance, evaluator variance, and actual behavior changes before deciding whether to rerun or block.

What telemetry would you collect to distinguish input drift, concept drift, and changes in an AI system's behavior?

Monitor input distributions, outcome relationships, and output behavior separately, using delayed labels or reviewed samples when correctness is not immediately observable.

Write a drift metric for categorical AI request distributions and explain how you would turn it into a useful alert.

Compute Jensen-Shannon divergence over a shared category vocabulary, then calibrate alerts against historical variability and minimum sample sizes.

Design observability for a retrieval-and-tool-using AI service without leaking sensitive prompts or creating unbounded metric cardinality.

Use structured traces for request-level diagnosis, bounded metrics for aggregation, and separately governed content sampling.

Instrument an asynchronous model call with latency and outcome telemetry while avoiding prompt capture and high-cardinality metric labels.

Wrap the call in a trace span and record duration and a bounded outcome label in a finally block.

An AI endpoint's median latency is stable, but p99 latency and cost per successful task are rising. How would you investigate?

Decompose request paths by stage, task slice, retries, and token usage; do not infer tail behavior from averages.

How would you define service-level objectives and release policies for an AI assistant whose semantic correctness is not immediately observable?

Separate operational SLOs from sampled quality indicators and enforce critical safety constraints independently.

Design a canary release for a new model and prompt combination serving customer-facing traffic.

Use stable randomized assignment, comparable cohorts, hard safety stops, and predefined exposure and decision rules.

Implement deterministic canary assignment that remains stable across application instances and supports gradual expansion.

Hash a stable assignment key with an experiment salt and compare the resulting bucket with the rollout percentage.

How would you test fallback behavior when a primary AI provider becomes unavailable or returns unusable output?

Define failure classes and fallback eligibility explicitly, then inject faults and verify deadlines, contracts, safety parity, and user-visible behavior.

What are the limitations of shadow testing an AI agent that can invoke tools, and how would you make shadow execution safe?

Mirror only approved data and replace side-effecting tools with isolated simulations or read-only adapters.

A provider silently updates a hosted model alias and your tool-call failure rate doubles. What is your incident response?

Contain exposure, preserve evidence, compare artifact and provider metadata, and restore a qualified path while isolating the cause.

Design a test architecture for a product combining deterministic business rules, retrieval, LLM generation, and external tools.

Test each contract independently, then use representative end-to-end evaluations to measure composition failures.

How would you build and govern an evaluation dataset that remains representative without leaking into prompt tuning?

Separate development and held-out sets, split by meaningful groups or time, and manage provenance, access, and dataset refreshes.

How would you define ownership and release decision rights across SDET, ML engineering, application engineering, security, and product teams?

Assign accountable owners for system risks and evidence, with explicit promotion authority and escalation paths.

How would you evaluate competing AI vendors for a production testing or customer-support platform?

Run a workload-specific bake-off covering quality, reliability, security, operations, and total cost under contractual constraints.

What governance controls would you require before releasing an AI feature that processes sensitive customer data?

Use risk classification, approved data flows, enforceable controls, release evidence, and ongoing accountability.

How would you measure the ROI of introducing AI-assisted test generation into an SDET organization?

Measure net business value against a credible baseline, including review effort, maintenance, infrastructure, and escaped-defect risk.

You inherit an SDET team with strong automation skills but little AI experience. What is your first 90-day strategy?

Baseline risks and capabilities, deliver one bounded pilot, and build reusable evaluation and operational practices around it.

Product wants to release an AI feature despite inconclusive quality results. How would you lead the decision?

Make the uncertainty and consequences explicit, then offer risk-bounded options rather than presenting an unsupported pass or fail.

How would you decide which AI outputs need human review and verify that the review process actually reduces risk?

Route by consequence and validated risk signals, then measure reviewer performance, workload, and missed failures.

Design a shared AI evaluation platform for multiple product teams without forcing every team into the same quality metric.

Standardize execution, provenance, security, and reporting while allowing domain-specific datasets, evaluators, and acceptance policies.

Design the quality and release architecture for a multi-tenant enterprise RAG assistant with strict document permissions.

Enforce identity-aware retrieval and evidence access, then evaluate retrieval, grounding, isolation, freshness, and end-to-end outcomes separately.

Design a safe autonomous agent that can propose and execute customer refunds, including its testing strategy.

Keep financial authority in deterministic policy services and treat the model's tool calls as untrusted proposals.

Implement asynchronous primary-to-fallback routing with a shared deadline and restricted fallback eligibility.

Give the primary a bounded share of the deadline, use only the remaining time for fallback, and propagate non-retryable failures.

Design an evaluation strategy for a multimodal system that extracts invoice data from images and initiates an approval workflow.

Evaluate perception, structured extraction, business validation, and workflow behavior as separate layers with end-to-end risk checks.

Design the test and observability strategy for a real-time voice assistant with interruption support and strict response-latency targets.

Measure the streaming pipeline end to end and test conversational state under partial transcripts, cancellations, and network faults.

Design a multi-tenant AI gateway that enforces spend limits while maintaining fairness, latency, and data isolation.

Combine authenticated tenant routing, atomic budget reservations, bounded concurrency, approved model policies, and usage reconciliation.

How would you secure and test the supply chain of an AI system that loads third-party models, evaluation plugins, and retrieval data?

Treat models, executable plugins, datasets, and indexes as separate supply-chain trust boundaries with verified provenance and constrained execution.

Implement reciprocal rank fusion for hybrid retrieval and describe how you would evaluate its effect on a RAG system.

Fuse ranked lists using reciprocal rank contributions, then test both retrieval relevance and downstream grounded-answer quality.

Design a reproducible evaluation harness for an agent whose behavior depends on time, external APIs, and mutable business state.

Control deterministic dependencies through versioned fixtures and stateful simulation, while measuring model variance separately.

A model rollout improves average task success but increases spend, causes occasional cross-tenant citations, and overloads human reviewers. How would you lead diagnosis and redesign?

Contain the isolation failure immediately, stabilize review and cost controls, then investigate each failure path before considering renewed rollout.