KEY POINTS
- METR's updated time horizon plot adds longer tasks and more open-ended, judgment-based ones.
- Models fell short of humans on designing AI evaluations and scoring philosophical arguments.
- Experts welcome the new tasks but say it is too early to predict progress on them.
The AI evaluation nonprofit METR has published an update to its widely shared time horizon plot, according to MIT Technology Review. The report finds that some AI capabilities continue to improve exponentially, while models appear less capable on tasks that require judgment than on coding.
The original plot rated models by the length of tasks they could complete, with each task's difficulty measured by how long human testers took to finish it. That version showed the task length that frontier models could handle doubling every four months. The chart was often misread because of the complexity of its measures.
The new version adds tasks that are estimated to take humans far longer than those in the first suite, along with more open-ended ones. The change matters because AI systems have become very good at verifiable work such as coding and math, where software can check automatically whether a solution works. Reinforcement learning, the technique used to train for such tasks, is harder to apply when success is poorly defined or slow to confirm, as with choosing a new research direction.
In one new task, models were asked to design AI evaluations that METR could use, and METR employees scored the results. Although the most advanced models now outperform almost every human at software engineering, none of the evaluations they designed was good enough for staff to consider useful. In another task, models scored philosophical arguments, and they again fell short of human performance. METR notes that its employees disagree on how relevant this task is to automating AI research and development.
There are signs of progress. Some of the newest models, such as Opus 5.5, tend to reach what METR calls borderline above the bar on the evaluation design tasks. A year ago, every model scored clearly below the bar.
Outside experts welcomed the broader approach. Sayash Kapoor, an incoming professor at UC Berkeley, said AI evaluations need to become more creative. He added that he was not surprised by the gap, pointing to an August preprint in which he and collaborators reported that models cannot yet reproduce human computer science research on their own. OpenAI also said in a Wednesday blog post that its models do worse on real tasks assigned by its employees than on the easier, verifiable tasks in METR's original suite.
The report has clear limits. Because the open-ended tasks are hard to design and score, few are included, which makes extrapolation difficult. Most completion times are now estimates, since paying people for tasks that may take hundreds of hours is prohibitively expensive. For the philosophy task, a single person rated most arguments. Najoung Kim of Boston University said she is not sure how the improvement would extrapolate, and METR's Matt Putz said clean trends are unlikely for judgment-laden tasks.
The update arrives as debate over automated AI research intensifies. An earlier, heavily redacted METR report claimed that AI might be doubling the pace of progress at Anthropic, which prompted fears of recursive self-improvement. Experts told MIT Technology Review that the new tasks are a step forward but that definitive answers remain far off.