Interview Questions

How do you add security scanning to a pipeline?

What token and session flaws do you test for?

Expired and tampered tokens, wrong audience or issuer, missing scope, algorithm confusion (alg: none), token reuse after logout, and refresh-token rotation.

How does testing an LLM-powered feature differ from testing a conventional rules-based feature?

Keep deterministic assertions for software contracts and use repeated, rubric-based evaluations for variable model behavior.

What are tokens and context windows, and how can they cause failures in an AI-assisted test-generation workflow?

Tokens are model-specific text units, and context limits constrain how much input and output a request can contain.

How would you test an embedding-based search service used to retrieve similar defects?

Measure retrieval relevance on labeled queries and separately verify filtering, indexing, and access-control behavior.

Does setting temperature to zero make LLM tests deterministic?

No. It reduces sampling variability but does not guarantee identical outputs across requests or environments.

How would you detect and reduce hallucinated APIs in AI-generated automation code?

Validate generated code against the repository's actual dependencies, then compile, execute, and review it.

Implement recall at k for a RAG retriever and explain what this metric cannot tell you.

Recall at k measures how many labeled relevant documents appear in the first k results; it does not measure answer correctness.

When would you choose prompting, RAG, or fine-tuning for an internal test-generation assistant?

Use prompting for task instructions, RAG for current repository knowledge, and fine-tuning for demonstrated recurring behavior gaps.

How would you design a defensible evaluation of an LLM that classifies test failures by root cause?

Use held-out, adjudicated incidents, repeated runs, per-class metrics, and explicit handling of uncertainty.

What controls would you require before sending production failure logs to an external LLM?

Minimize and sanitize data, verify provider handling terms, enforce access controls, and audit the complete data path.

Why is a tool-calling test agent riskier than a text-only assistant, and where should controls live?

Tool calls can change systems, so authorization and validation must be enforced outside the model.

How would you qualify a provider model upgrade for an AI testing platform?

Run paired offline evaluations, verify integration contracts, then use a monitored canary with rollback capability.

What information should a prompt contain to generate maintainable automated tests from a user story?

Provide verified requirements, framework constraints, repository conventions, expected assertions, and a clear output contract.

How would you select few-shot examples for a prompt that converts defects into regression tests?

Choose correct, diverse examples that demonstrate the desired reasoning evidence and test conventions without leaking evaluation cases.

Implement a testable prompt builder that keeps trusted instructions separate from an untrusted defect report.

Build separate messages, serialize the report as data, enforce input limits, and test message construction deterministically.

How should a test-generation assistant handle conflicting requirements, examples, and repository conventions?

Use an explicit source-precedence policy and surface unresolved conflicts instead of silently choosing a convenient interpretation.

How would you prompt an LLM to discover useful edge cases without producing a long list of redundant tests?

Provide explicit boundaries and ask for risk-ranked equivalence classes with requirement references.

How can metamorphic testing evaluate a prompt when there is no single correct answer?

Define justified relationships between related inputs and check whether outputs preserve those relationships.

Validate structured LLM-generated test plans and distinguish schema validation from semantic validation.

Parse against a strict schema, then check domain rules such as unique identifiers and valid requirement references.

How would you test indirect prompt injection through repository files or retrieved documentation?

Place malicious instructions in untrusted sources and verify both model behavior and independently enforced action boundaries.

How would you version and regression-test prompts used in a shared SDET platform?

Treat prompts as versioned production artifacts and evaluate changes against fixed, representative cases before release.

An assistant ignores a requirement even though it was included in the request. How would you debug the prompt pipeline?

Inspect the actual assembled request, verify context integrity, and isolate competing instructions before rewriting the prompt.

What should you request from an LLM to make a failure-triage recommendation auditable?

Request a concise conclusion, supporting evidence, uncertainty, alternatives, and next verification steps.

Design a safe CI workflow for AI-generated test changes.

Generate a reviewable patch in an isolated environment, run layered checks, and require approval before merging.

An LLM generates a flaky Playwright cart test. Rewrite it to control backend data and assert user-visible behavior.

Use scoped network fixtures, accessible locators, and retrying assertions instead of arbitrary sleeps.

What would you review in AI-generated tests beyond whether they pass?

Review test intent, oracle independence, failure sensitivity, isolation, security, and maintainability.

How would you use AI to diagnose flaky tests without allowing it to hide real product defects?

Use AI to rank evidence-backed hypotheses, then verify them with controlled experiments before changing tests.

Implement bounded retries for an LLM API adapter and explain how to test them deterministically.

Retry only classified transient failures, cap attempts, and inject the operation and sleeper for deterministic tests.

Should an AI self-healing locator automatically replace a broken selector in production test suites?

It may propose a replacement, but semantic equivalence and test intent must be verified before accepting it.

Implement metrics for a judge that decides whether generated tests are acceptable, and explain the danger of false acceptance.

Compare judge decisions with adjudicated labels and track false acceptance separately from aggregate accuracy.

How would you validate AI-based test selection before allowing it to skip tests in CI?

Run it in shadow mode against full-suite outcomes and measure missed failures, not just runtime savings.

How would you use AI to generate API tests from an OpenAPI specification without overtrusting the specification?

Generate contract-focused cases from the schema, then supplement them with business invariants and independently reviewed security scenarios.

Write a deterministic UI test for an AI assistant's abstention response and explain where live-model evaluation belongs.

Mock the model-facing application API for the UI test and evaluate real abstention behavior in a separate suite.

How would you performance-test an LLM-backed failure-analysis service while controlling cost and avoiding misleading results?

Test infrastructure with controlled substitutes, then run bounded live load tests with representative token sizes and provider limits.

Design an AI agent that investigates a failed build and proposes a test fix without independently weakening quality controls.

Separate evidence collection, proposal generation, sandbox validation, and human approval behind explicit policy boundaries.

How would you lead a rollout of AI-assisted testing and demonstrate value without incentivizing low-quality test generation?

Start with a bounded pilot and measure verified engineering outcomes, review burden, and safety—not generated test counts.

How would you design a test strategy for a RAG assistant without treating every failure as a model-quality problem?

Test ingestion, retrieval, context assembly, generation, and application controls separately, then connect them with end-to-end scenarios.

What makes a golden evaluation dataset trustworthy and maintainable for an enterprise RAG system?

Use representative, versioned cases with evidence-backed expectations, explicit rubrics, and protected holdout data.

Implement retrieval metrics for ranked document IDs and explain what recall@k, precision@k, and reciprocal rank fail to measure.

Measure evidence coverage and ranking separately; retrieval metrics alone do not establish answer quality.