Engineering at Anthropic dropped another banger. Their internal playbook for evaluating AI agents. Here's the most counterintuitive lesson I learned from it: Don't test the steps your agent took. Test what it actually produced. This goes against every instinct. You'd think checking each step ensures quality. But agents are creative. They find solutions you didn't anticipate. Punishing unexpected paths just makes your evals brittle. What matters is the final result. Test that directly. The playbook breaks down three types of graders: - Code-based: Fast and objective, but brittle to valid variations. - Model-based: LLM-as-judge with rubrics. Flexible, but needs calibration. - Human: Gold standard, but expensive. Use sparingly. It also covers eval strategies for coding agents, conversational agents, research agents, and computer use agents. Key takeaways: - Start with 20-50 test cases from real failures - Each trial should start from a clean environment - Run multiple trials since model outputs vary - Read the transcripts. This is how you catch grading bugs. If you're serious about shipping reliable agents. I highly recommend reading it. Link in the next tweet.