DeepRSI

Introduction

DeepRSI is the first of many benchmarks, evals, and other research contributions we plan to release to help the industry test, measure, and hillclimb various capabilities needed for recursively self-improving AI. This benchmark contains a batch of 15 open-ended machine learning problems that measure how much an AI agent can optimize/improve on a given baseline over 48 hours.

While benchmarks like PostTrainBench¹ and RSI Bench² solely focus on language models, DeepRSI includes a more general set of problems as we wanted to showcase a broader picture of what improvements agents can make. We include tasks in the following categories:

• Deep learning training. Changing how a model is trained, so it does better on held-out data (e.g., Chess LM Pretraining Recipe).
• Speedup and kernels. Making existing GPU code run faster while keeping results the same (e.g., FlashMLA FP8 Sparse Attention).
• Estimator Improvement. Improving how accurately a model or estimator predicts on held-out data (e.g., Severity-Agnostic Turbulence Mitigation).
• Algorithm optimization. Improving search, compression, and denoising programs (e.g., SIFT1M 32-MiB Vector Search).

Methodology

Each model is run twice with their highest reasoning effort, the Pi agent harness³, no network access, and a 48 hour time budget.

Sourcing tasks from real research. At hillclimb, we collaborate with highly qualified researchers from which we observe real environments they face while working on state-of-the-art research. Given our privileged access to novel work, we’re able to derive tasks that are much more closely aligned to real problems and bottlenecks that define the frontier. Unlike existing contributor-based benchmarks, improved agent performances on DeepRSI will signify a verifiable lift in self-improving capabilities that would actively progress ongoing research.

Initial instructions. Agents are told what they need to optimize, how to test their work, and everything they would need to properly work on this task. We do not explicitly tell the agent to work or experiment a certain way and leave its research loop up to their judgement. In certain examples, we describe a given behavior as off limits.

Auto-resume. Our tasks are open-ended and do not have a tight maximal upper bound. Through smaller-scale trials we continue to observe consistent increases in these tasks with increasing time budgets less than 48 hours. For this reason, we believe ending early on these tasks leads to leaving room for improvement on the table that can still be hillclimbed with the remaining time budget.

Frontier models are getting better at using the full time budget, but we still find most models will end significantly early. To combat this, we auto-resume the session if the agent ends early with a short continuation prompt to continue its work and use as much time as possible. There were some cases for a few models where the agent auto-resumes only to “give up” by submitting its last submission in rapid succession. In these cases, we chose to end the trial early.

Scoring. Each task comes with a baseline threshold which agents are expected to improve on. For each attempt, we measure the agent’s improvement over that baseline. Given that tasks differ in difficulty, we compute Elo scores to compare relative agent performance fairly as follows:

1. On each task, each model's attempt goes head to head with every other model's.
2. Fit model strength with a Bradley–Terry model⁴, which gives each model an Elo rating such that P(i beats j) = 1 / (1 + 10^{(R_j − R_i) / s}), where R_i is the Elo rating of model i and s is the rating scale.

For future submissions, we plan to freeze the Elo of the original models tested, and score future submissions against them.

Public and private verifiers. For each task, there is a public verifier that the agent can run as often as it wants to evaluate its work. Every time the agent runs the public verifier, a snapshot of the coding environment is saved as a checkpoint. After the attempt ends, a separate private verifier grades every checkpoint and the final code. For most tasks, it grades on held-out data the agent never sees, such as unseen puzzles, mazes, queries, or videos. Only the private verifier’s grade counts toward the recorded score. This way, the public verifier serves as a tool for real-time feedback as the agent hillclimbs the task, while the private verifier allows us to accurately evaluate the performance of the agent, protected from overfitting and reward hacking.

Dealing with reward hacking. For this benchmark, the agents are given no network access. We note this as a possible limitation for autoresearch-style tasks. We're working with various research partners to release future variants where the agent can access the internet with proper guardrails.

Each task was also checked to have reasonably strong and difficult-to-hack verifiers. We list some of these later in the post, including ones we saw in early internal testing and potential hacks we found through our own agentic red teaming.

Results

Overall Model Rankings

Each model is ranked by its performance at the end of the 48-hour mark. Astra takes an early lead over Fable, though as tasks continue longer, Fable appears to have better ideas and exploration, leading to the best final performance.

The figure above shows each model’s final private-verifier Elo across all task types. Fable leads in all categories, with an especially large lead for deep learning training and estimator improvement tasks. Astra follows closely behind, decisively scoring 2nd in all categories. Grok and Gemini trade blows on different task types, most notably with Gemini doing significantly better on speedup and kernel tasks.

Limitations

Given that our sandboxes had 24-hour limits, we implemented a workaround to rotate sandboxes and transfer filesystem snapshots at the 24-hour mark. During that time, any running processes (e.g., training) are stopped, and in the new sandbox the agent is instructed to check if it needs to restart anything. While this solution helped us deal with sandbox limitations, we are actively working on better infra to properly evaluate and train on ultra-long-horizon tasks.

Due to our tasks having 48-hour-long horizons, we could not control for agent behavior that caused large variances in token usage, sometimes even to the detriment of its own performance:

Polling. Agents sometimes decided to poll instead of setting up a wakeup signal, causing much higher usage.

Hangs. Agents often launched background processes and awaited their results. In the few cases where these background processes never returned, the agent would wait forever. Fable and Astra were often the only agents to circumvent this by setting timeouts.

Work Variance. Even in the same task, two different attempts by the agents could exhibit drastically different token usage, as one run could involve longer-running sweeps, resulting in fewer output tokens.

Future work

Our immediate next objective is to experiment with substantially longer time horizons.

From our testing, we’ve found evidence to suggest that our tasks do not saturate at 48 hours, and that there is still room for improvement on even longer horizons. We attribute this to the fact that we source sophisticated, realistic environments from production traces that challenge frontier agents.

We also want to test models in their native harnesses, like GPT with Codex and Claude with Claude Code. Separately, we want to explore specialized internet access that enables research agents to connect insights across papers and other external sources, rather than simply regurgitating static knowledge.

Finally, while autoresearch tasks are great at measuring the raw long-horizon intelligence of these models, true recursively self-improving AI will require other capabilities that are less verifiable (e.g., research taste). In the future, we plan to release benchmarks, evals, and other research contributions that will help reflect other parts of the capabilities needed for RSI.

Acknowledgements

We thank the key collaborators who helped Hillclimb build DeepRSI:

• Khurram Yamin
• Juliana Li
• Pingbang Hu

References

1. Ben Rank, Hardik Bhatnagar, Ameya Prabhu, Shira Eisenberg, Karina Nguyen, Matthias Bethge, and Maksym Andriushchenko. PostTrainBench: Can LLM Agents Automate LLM Post-Training? In International Conference on Machine Learning (ICML), 2026. arXiv:2603.08640. arxiv.org/abs/2603.08640
2. Anas Mahmoud, MohammadHossein Rezaei, Mingchen Zhuge, Chenguang Wang, Yuandong Tian, and Yunzhong He. Announcing RSI Bench. Scale AI, August 2026. rsi-benchmark.com/blog/announcing-rsi-bench
3. Mario Zechner and Earendil. Pi: a minimal agent harness. Software, 2026. pi.dev
4. Bradley–Terry model. Wikipedia. en.wikipedia.org/wiki/Bradley–Terry_model

Citation

@misc{hillclimb2026deeprsi,
  title  = {DeepRSI},
  author = {Rajaghatta, Pruthvi and Devarinti, Pranav and Berezovsky, Alex and Yamin, Khurram and Li, Juliana and Hu, Pingbang and Park, Jun},
  year   = {2026},
  url    = {https://www.deeprsi.com}
}