Hard20 minAI Engineering
UpdatedAug 6, 2026
Edit

Agent: Debug a Bad Outcome

Question Variations

  • "What should an agent trace contain?"
  • "How would you turn an incident into an evaluation case?"

Why This Is Asked

An agent returns the wrong answer after several tool calls. Design traces and evaluation so an engineer can identify whether planning, retrieval, tool selection, arguments, or tool output caused the failure.

Key Concepts

  • Trace: Record model turns, tool calls, tool results, state transitions, and timings.
  • Privacy: Redact secrets and apply access controls to traces.
  • Labels: Classify failure stages to guide fixes and regression tests.
  • Evaluation: Replay representative tasks and score end-to-end outcomes.

Question Variations

  • “What should an agent trace contain?”
  • “How would you turn an incident into an evaluation case?”

Answers by Technology

+ Add Variant
System DesignImprove this answer ✏️

Expected Answer

Trace model turns, state changes, tool selection, arguments, results, timing, and policy decisions using a correlation ID. Redact secrets and restrict trace access. Label incidents by retrieval, planning, arguments, tool output, or execution, then turn each into a replayable evaluation case.

Why It Matters

Without traces, an agent failure is an unverifiable story rather than a fixable defect.

Common Mistakes

  • Logging secrets: Observability becomes a data breach.
  • Recording only final output: The failing step cannot be located.

Follow-up Questions

  • What makes a useful trace? (Answer: Causal steps, inputs, outputs, and timing.)
  • How does an incident improve quality? (Answer: It becomes a regression evaluation.)