My Coding Agent Passed the Eval. It Shouldn’t Have.
My coding agent passed the eval. The problem was, it shouldn’t have.
While building Builder-Bench, an open-source eval for coding agents for product management tasks, I found cases where the LLM judge was giving agents credit for work it had no way to verify. In one case, the judge could see that the agent ran a verification command, but not the command’s output. So when the agent said “verified identical,” the judge gave it a pass. It wasn’t checking the work; it was trusting the agent’s account of what happened.
That sent me down a rabbit hole. I hand-labeled transcripts blind, compared my labels against the judge, and found four different ways the grader could get things wrong. Across four rounds, human–judge agreement improved from Cohen’s κ=0.40 to 0.93 on the final batch as I fixed the rubric, changed what evidence the judge could see, and added regression tests so the same failures wouldn’t come back.
This talk walks through the eval failures, the transcripts that exposed them, and the debugging loop that emerged: grade blind, measure agreement, inspect the disagreements, fix the rubric or the evidence, and test again.
The broader lesson is simple: a coding agent can look like it passed when the real failure is in the LLM judge. We need a systematic way to identify them and fix them.
About the speaker
Helena Tan
Product Lead, AI Agent at Box
Helena Tan is a Product Manager with about eight years on ML and AI products, previously at Niantic, Uber, and Fitbit, and now building AI Agent for Enterprise Customers at Box. On her own time she builds with coding agents and measures them: Builder-Bench, an open-source eval of Claude Code on PM work with a judge validated against 142 hand labels, and the PM-AI Hiring Index, a daily data product she built end to end with Claude Code. She writes about what she learns at helenatan.substack.com