Schedule

Code agents have made remarkable progress in the past few months, and much of that progress is directly due to scale: more compute and data have saturated benchmarks once considered nearly impossible. It could be tempting to believe that scaling is the only thing that matters and that nothing can be achieved by small teams with limited compute. Yet, code agents remain surprisingly fragile, sometimes failing on seemingly easy issues, or producing code that falls short of production quality standards.

In this talk, we will first explore how scaling is done in practice at companies such as Mistral. Afterwards, we will discuss the types of issues that pure scaling will not solve and where resource-constrained teams are well positioned to make impactful contributions.

When an agent fails, was it the planning or the interface? TALES tests this with 122 text-adventure tasks, one minimal prompt for every model, and detectors that attribute failures to specific skills. While synthetic text worlds are nearly solved, human-written games are still providing some challenges. Interestingly, we found that coding agents show the same failures: repeating failed commands, acting on stale state, losing the overall goal.