Interview Questions

How would you detect ingestion and chunking regressions before they appear as incorrect RAG answers?

How would you evaluate groundedness when an answer combines supported facts with plausible but unsupported conclusions?

Decompose the answer into verifiable claims and assess whether each claim is supported by the supplied evidence.

How do answer correctness, groundedness, relevance, and completeness differ in an LLM evaluation?

They measure truth or task validity, evidence support, responsiveness, and coverage of required information, respectively.

Write a validator for an LLM-generated refund decision that enforces JSON structure and business rules.

Validate strict JSON, schema constraints, and cross-field business invariants before consuming the output.

How would you test citation quality beyond checking whether an answer contains links?

Check citation validity, claim support, coverage, provenance, and user authorization.

How would you evaluate whether a RAG assistant answers, abstains, or asks for clarification at the right time?

Label answerability and ambiguity explicitly, then measure both unsafe answers and unnecessary abstentions.

How do you make LLM regression tests useful when repeated executions can produce different answers?

Test behavioral invariants, repeat representative cases, and report distributions rather than requiring identical text.

Implement a metamorphic test for a RAG assistant when a complete reference answer is unavailable.

Apply transformations with known semantic effects and assert task-specific invariants rather than identical wording.

How would you test a multi-turn assistant's memory, corrections, and context-window behavior?

Evaluate conversation trajectories, including state updates, explicit corrections, topic changes, and context truncation.

How would you evaluate a multilingual RAG assistant without assuming translated English tests are sufficient?

Use native-language cases, locale-specific expectations, and cross-language retrieval tests with per-language reporting.

How would you validate an LLM-as-judge before using its scores in release decisions?

Calibrate it against independently labeled human examples and measure its errors, biases, and stability.

Implement a paired bootstrap confidence interval for the quality difference between two systems evaluated on the same cases.

Resample case-level paired differences and estimate uncertainty around the mean candidate-minus-baseline score.

Design release gates for an LLM application that balance quality, safety, latency, and evaluation cost.

Use staged gates with deterministic contracts, targeted behavioral suites, statistical quality checks, and non-negotiable safety boundaries.

Why can a higher aggregate evaluation score hide a serious regression, and how would you prevent that?

Report meaningful slices, stable weighting, sample sizes, and critical failure counts alongside the overall score.

How would you build a reliable human annotation workflow for ambiguous LLM answers?

Use explicit rubrics, independent overlapping reviews, evidence access, adjudication, and annotation-quality monitoring.

Write tests proving that a tool-execution boundary rejects unauthorized model-generated calls before any side effect occurs.

Treat tool calls as untrusted proposals and verify deny-before-execute behavior using server-side identity and permissions.

How would you test tool-calling behavior under timeouts, retries, partial failures, and malformed tool responses?

Inject failures at execution boundaries and verify bounded retries, idempotency, reconciliation, and truthful user-facing outcomes.

How would you evaluate an agent that can achieve a task through several different valid tool sequences?

Score verified outcomes and execution constraints rather than requiring one exact reference trajectory.

How would you performance-test a streaming RAG application while preserving visibility into answer quality?

Measure end-to-end and stage-level latency, streaming behavior, throughput, errors, and cost on a representative workload.

What observability would you add so an SDET can diagnose and replay an LLM failure without overcollecting sensitive data?

Capture versioned execution metadata and privacy-controlled artifacts linked by a trace ID.

How would you test resistance to direct prompt injection without confusing harmless user instructions with security violations?

Define protected behaviors and trust boundaries, then measure whether adversarial user input causes a prohibited outcome.

Implement an indirect prompt-injection regression test for instructions embedded in retrieved content.

Place a controlled malicious instruction in a relevant document and inspect both the answer and trusted execution trace.

How would you test tenant isolation across retrieval, caching, conversation memory, and tool execution?

Seed distinguishable tenant data and verify isolation at every stateful and authorization boundary.

How would you assess sensitive-data leakage without using real credentials or customer records in your tests?

Use synthetic secrets and privacy fixtures, then inspect every output and persistence channel.

How would you test an agent's resistance to data exfiltration and server-side request forgery through network tools?

Test enforced destination and payload policies using controlled endpoints, with the model treated as an untrusted caller.

How would you security-test model-generated JSON, HTML, Markdown, SQL, or shell-like content consumed by downstream systems?

Treat generated output as untrusted input and test the sink-specific controls that prevent execution or unsafe interpretation.

How would you evaluate bias in an AI assistant while avoiding simplistic demographic score comparisons?

Combine task-specific harm definitions, controlled counterfactuals, representative subgroup cases, and uncertainty-aware analysis.

How would you test resilience to poisoned documents and knowledge-base updates in a RAG system?

Inject controlled misleading content into a test corpus and evaluate ingestion trust, retrieval influence, provenance, and recovery.

Implement a synthetic-canary scanner for leakage in user responses, tool payloads, and logs, and explain its limits.

Generate a unique fake marker and scan serialized artifacts for exact and explicitly supported encoded forms.

How would you organize an authorized red-team program for an enterprise AI agent?

Start with threat models and rules of engagement, then turn verified findings into mitigations and regression tests.

How would you monitor production AI quality and detect drift without assuming offline evaluation predicts every failure?

Combine operational signals, privacy-aware sampled evaluation, user outcomes, and version-attributed change detection.

A candidate model improves the automated score but domain experts identify new serious errors. How would you lead the release decision?

Pause the decision, investigate disagreement at the case level, and apply predeclared risk criteria rather than choosing the most favorable metric.

How would you version an AI application's model, prompts, retrieval configuration, and evaluation data so a production result can be traced to a release?

Create an immutable release manifest that references every behavior-changing artifact and record its identifier with each execution.

How would you structure CI quality gates for an LLM feature without making every pull request slow and expensive?

Use risk-based test tiers: deterministic checks on every change, targeted evaluations on pull requests, and broader statistical evaluations before promotion.

Implement a paired statistical quality gate that rejects a candidate when non-inferiority to the baseline has not been demonstrated.

Compute paired score differences and require their one-sided lower confidence bound to exceed a predefined non-inferiority margin.

How would you distinguish AI evaluation flakiness from a genuine quality regression in CI?

Separate infrastructure failures, generation variance, evaluator variance, and actual behavior changes before deciding whether to rerun or block.

What telemetry would you collect to distinguish input drift, concept drift, and changes in an AI system's behavior?

Monitor input distributions, outcome relationships, and output behavior separately, using delayed labels or reviewed samples when correctness is not immediately observable.

Write a drift metric for categorical AI request distributions and explain how you would turn it into a useful alert.

Compute Jensen-Shannon divergence over a shared category vocabulary, then calibrate alerts against historical variability and minimum sample sizes.

Design observability for a retrieval-and-tool-using AI service without leaking sensitive prompts or creating unbounded metric cardinality.

Use structured traces for request-level diagnosis, bounded metrics for aggregation, and separately governed content sampling.

Instrument an asynchronous model call with latency and outcome telemetry while avoiding prompt capture and high-cardinality metric labels.

Wrap the call in a trace span and record duration and a bounded outcome label in a finally block.