Advanced project
MCP-connected agent with state-based evals
Builds on 3 lessonsabout 1 h 20 min of reading first
In this project
What you need first
- Completed:Model Context Protocol and tool ecosystemsTools and agents
- Completed:Multi-agent systems: when to use them and what they costContext, memory and multi-agent systems
- Completed:Agent evaluation: state, trajectory, pass^k and release gatesContext, memory and multi-agent systems
Skills you will practise
- Tool calling
- Model Context Protocol
- Multi-agent systems
- Agent evaluation
Problem
Build an agent that uses a tool server over MCP and measure how reliable it is.
Requirements
- An MCP server with one tool and one resource.
- A host or agent that uses it, with step and token budgets.
- Human approval on write actions.
- 15 tasks graded by end state, run k=5 times.
Prerequisites
- Model Context Protocol and tool ecosystems
- Multi-agent systems: when to use them and what they cost
- Agent evaluation: state, trajectory, pass^k and release gates
Tasks
- To do: Build the MCP server (1 tool, 1 resource).
- To do: Connect a host or agent to it.
- To do: Add step and token budgets and approval on writes.
- To do: Write 15 tasks with state-based graders.
- To do: Run each task 5 times and report pass@k and pass^k.
- To do: Write a short threat model.
Evaluation
Record real measurements and never invent numbers. The finished project should give you: agent design, a security threat model and reliability metrics.