Lesson deck
Agent evaluation: state, trajectory, pass^k and release gates flashcards
Reviewing needs JavaScript. You can read every card in the list below.
Loading your cards...
Back to the lesson3 cards in this deck
Read all 3 cards as a list
- Front
- What to trace per agent step?
- Back
- Prompt version, chosen tool and arguments, result, tokens, latency, cost and errors: enough to replay a failure.
- Front
- pass@k vs pass^k, and which fits a one-shot agent?
- Back
- pass@k: at least one of k runs succeeds. pass^k: all k succeed. A one-shot agent needs pass^k, which exposes the run-to-run variance that pass@k hides.
- Front
- Why grade an agent by state, not by its final message?
- Back
- The message is a claim and can be false. Check the world afterwards (the database row, the passing tests, the file), then use trajectory metrics for cost and risk.