During the last day of their “shipmas” event, OpenAI covered information about their next AI model “o3”, due for initial public release starting at the end of January 2025. Initial release will be “o3-mini”, the least powerful of the new models but much faster and more cost efficient while still outperforming current top models. Later will be the full o3 model.

While the results shown here are from the later full o3 model using an aggressive amount of processing that won’t be available to the public at first, this will still show what to expect in the next few months from publicly available AI tools.

Math

The o3 model scored 96.7% accuracy on the AIME 2024 math competition, missing only one question. Success in the AIME requires a deep understanding of high school mathematics, including algebra, geometry, number theory, and combinatorics. Performing well on the AIME is a significant achievement, reflecting advanced mathematical abilities.

The o3 model also solved 25.2% of problems on EpochAI’s Frontier Math benchmark. For reference, current AI models (including o1) have been stuck around 2%. FrontierMath, developed by Epoch AI, is a benchmark comprising hundreds of original, exceptionally challenging mathematics problems designed to evaluate advanced reasoning capabilities in AI systems. These problems span major branches of modern mathematics—from computational number theory to abstract algebraic geometry—and typically require hours or days for expert mathematicians to solve.

GPQA Diamond Subset

The GPQA Diamond subset is a highly challenging component of the GPQA (Graduate-Level Google-Proof Q&A) benchmark, designed to assess advanced scientific reasoning in AI systems. This subset comprises 198 multiple-choice questions across biology, physics, and chemistry, each crafted by domain experts to require deep understanding and multi-step reasoning.

Expert-Level Difficulty: Even individuals with or pursuing PhDs in the relevant fields achieve approximately 65-70% accuracy on these questions, underscoring their complexity.

Google-Proof Design: The questions are constructed to resist simple web searches, necessitating genuine comprehension rather than superficial information retrieval. Skilled non-experts, despite having unrestricted internet access, attain only about 34% accuracy, highlighting the benchmark’s rigor.

The o3 model scored 87.7%.

Codeforces Competitive Coding

Codeforces is a prominent platform for competitive programming, hosting a wide range of contests that test problem-solving and algorithmic skills. It’s widely used by programmers to improve their skills, compete with peers globally, and prepare for competitive programming competitions like ACM-ICPC or IOI.

Codeforces uses a rating system inspired by the Elo system, originally designed for chess, but adapted for competitive programming. This system ranks users based on their contest performance.

The o3 model currently has a rating of 2727 on Codeforces, setting it in the “International Grandmaster” category and in the top 200 humans in the world that have competed.

ARC-AGI Benchmark

The ARC-AGI Benchmark was covered in a recent post here, and as of last month a new AI reasoning method had reached a significant milestone of >60% completion.

The o3 model scored 87.5%, surpassing average human performance on this conceptual reasoning test.

Conclusion

OpenAI is far from the only company pushing incredible advancements in their models on an ever shorter timeframe. Even disregarding competitors, it is expected that newer frontier models beyond the one highlighted above will be released every 3 months next year just by OpenAI. Remember this whenever you see articles that talk about how AI has “hit a wall” in advancement.