AI/ES/25/11 Code/CODE/21 Fri/1010 From Code Snippets to Codebases Then Now and What's Next for Coding Evals
10:10am - 10:30am Talk: From Code Snippets to Codebases: Then, Now, and What's Next for Coding Evals Person/Naman Jain
Talk: From Code Snippets to Codebases: Then, Now, and What's Next for Coding Evals
- AI coding capabilities have leapt from generating one-line snippets to competing entire codebases with agentic workflows. I'll trace that arc focusing on learnings and challenges through each stage. I will start with early testable coding benchmarks distilling lessons about contamination and distributional overfitting. Next, moving beyond isolated programming problems, I will talk about repository grounded coding problems from SWE-bench style bug fixing, and R2E's automated function completion setting. We'll then move beyond isolated functions to longer-horizon tasks—runtime optimization (GSO), translation (Syzygy), and refactoring—highlighting challenges like test hacking, code quality, and idiomaticity. Finally, beyond code generation, I will talk about human preference evaluation in chatting (LMArena RepoChat) and developer-preference signals in-IDE via Copilot Arena.