UpdatedAug 6, 2026

Agent: Debug a Bad Outcome

Question Variations

"What should an agent trace contain?"
"How would you turn an incident into an evaluation case?"
"An agent gave a wrong answer after several tool calls. How would you isolate the failed stage and prevent recurrence?"

Why This Is Asked

This tests whether a candidate can make multi-step agent failures observable and reproducible. Interviewers assess causal tracing, privacy controls, failure taxonomy, and converting incidents into regression evaluations.

Key Concepts

Trace: Record model turns, tool calls, tool results, state transitions, and timings.
Privacy: Redact secrets and apply access controls to traces.
Labels: Classify failure stages to guide fixes and regression tests.
Evaluation: Replay representative tasks and score end-to-end outcomes.

Question Variations

“What should an agent trace contain?”
“How would you turn an incident into an evaluation case?”
“An agent gave a wrong answer after several tool calls. How would you isolate the failed stage and prevent recurrence?”

Answers by Technology

+ Add Variant

System DesignImprove this answer ✏️

Expected Answer

I trace each model turn, state transition, selected tool, validated arguments, result, policy decision, and timing under one correlation ID. Traces are redacted and access controlled. When an incident occurs, I label the failing stage—retrieval, planning, argument construction, tool result, or execution—then add a sanitized replay case to the evaluation suite. That turns incidents into measurable regressions instead of anecdotes.

Why It Matters

Without traces, an agent failure is an unverifiable story rather than a fixable defect.

Common Mistakes

Logging secrets: Observability becomes a data breach.
Recording only final output: The failing step cannot be located.

Follow-up Questions

What makes a useful trace? (Answer: Causal steps, inputs, outputs, and timing.)
How does an incident improve quality? (Answer: It becomes a regression evaluation.)