> ## Documentation Index > Fetch the complete documentation index at: https://hydroxai.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Humanity's Last Exam > The hardest multi-domain academic benchmark — 3,000 expert-crafted questions across 100+ disciplines designed to be the final exam before AGI. Humanity's Last Exam ## Overview **Humanity's Last Exam (HLE)** is a crowdsourced benchmark of approximately 3,000 questions spanning over 100 academic disciplines, created by domain experts from around the world. It was designed with a provocative mission: to be the **last exam humanity would need to create** — one so difficult that only a truly general intelligence could pass it. Created by the Center for AI Safety (CAIS) and Scale AI, Humanity's Last Exam represents the most ambitious attempt to test the frontier of AI knowledge and reasoning across the full breadth of human expertise. ## Key Details | Property | Value | | ------------------ | -------------------------------------- | | **Created by** | Center for AI Safety (CAIS) & Scale AI | | **Released** | January 2025 | | **Task type** | Expert-level academic questions | | **Dataset size** | \~3,000 questions | | **Domains** | 100+ academic disciplines | | **Question types** | Multiple-choice and short-answer | | **Evaluation** | Accuracy (% correct) | ## How It Works 1. **Expert contribution**: Thousands of researchers and domain experts submitted their hardest questions — problems that would challenge even PhD-level specialists 2. **Curation**: Questions were filtered for quality, uniqueness, and difficulty 3. **Format**: Mix of multiple-choice and short-answer questions 4. **Evaluation**: Models are tested zero-shot with standard prompting ### Example Domains The benchmark spans an extraordinary breadth of human knowledge: | Domain Cluster | Example Subjects | | ------------------- | ---------------------------------------------------------------------------- | | **STEM** | Quantum mechanics, algebraic topology, organic chemistry, genomics | | **Humanities** | Classical philology, art history, medieval literature, philosophy of mind | | **Social Sciences** | Behavioral economics, political theory, linguistics, anthropology | | **Professional** | Clinical medicine, constitutional law, patent examination, actuarial science | | **Specialized** | Entomology, numismatics, paleoclimate reconstruction, music theory | ## Why It Matters Humanity's Last Exam addresses a critical problem in AI evaluation: **benchmark saturation**. As models quickly approach ceiling performance on existing benchmarks (MMLU, GPQA, etc.), the field needs harder tests that can differentiate frontier capabilities. * **Expert-caliber difficulty** — Questions are designed to be challenging even for specialists in each field * **Breadth over depth** — Tests the generality of knowledge, not just narrow expertise * **Contamination-resistant** — Novel questions not found in training data * **AGI milestone** — Performance on HLE is widely tracked as a proxy for progress toward general intelligence ## Notable Results | Model | Accuracy | Date | | ----------------- | -------- | ---- | | OpenAI o3 | \~26% | 2025 | | Claude 3.5 Sonnet | \~18% | 2025 | | Gemini 2.0 Ultra | \~22% | 2025 | | GPT-4o | \~15% | 2025 | | DeepSeek-V3 | \~16% | 2025 | Humanity's Last Exam remains far from solved. The best models get approximately 1 in 4 questions correct, highlighting the vast gap between current AI and comprehensive human-level expertise. ## Scoring Breakdown Performance varies dramatically across domains: | Domain | Best Model Score | Observation | | ----------------- | ---------------- | ------------------------------------------------------------- | | Mathematics | \~35% | Models perform relatively better on formal reasoning | | Physics | \~30% | Strong on computational problems, weak on conceptual ones | | Biology | \~20% | Specialized knowledge gaps are evident | | Humanities | \~12% | Nuanced cultural and interpretive knowledge remains very hard | | Niche specialties | \~8% | Rare domains expose the limits of training data breadth | ## Limitations * **Static dataset** — Will become contaminated over time as questions leak into training data * **Expert bias** — Questions reflect the priorities of contributing researchers, not uniform coverage * **Format constraints** — Multiple-choice and short-answer don't test open-ended reasoning or synthesis * **No partial credit** — Binary scoring doesn't capture "close" answers or sound reasoning with wrong conclusions ## References * [Humanity's Last Exam](https://lastexam.ai) — Official website * [CAIS Announcement](https://www.safe.ai/humanitys-last-exam) — Center for AI Safety blog post * [Paper](https://arxiv.org/abs/2501.14249) — Technical description and analysis