Evaluating Large Language Models in Scientific Discovery
A recent study introduces the Scientific Discovery Evaluation (SDE) framework to assess large language models on tasks that more closely resemble actual scientific research. Unlike existing science benchmarks that largely test decontextualized knowledge, SDE evaluates models across biology, chemistry, materials science, and physics using research projects defined by domain experts. These projects are broken down into modular research scenarios, from which vetted questions are sampled.
The framework operates at two levels. At the question level, it measures accuracy on scenario-tied items. At the project level, models must propose testable hypotheses, design or execute simulations or experiments, and interpret the results.
Benchmark composition
SDE contains 1,125 questions linked to 43 research scenarios (chemistry: 276 questions across 12 scenarios; materials: 486 questions across 13 scenarios; biology: 200 questions across 10 scenarios; physics: 163 questions across 8 scenarios). Questions were collected through a hybrid process that combined semi-automated sampling from open datasets with manual expert curation, followed by panel review for validity. Evaluation formats include multiple-choice, exact match, tolerance-based scoring, and similarity metrics such as Tanimoto similarity for chemical structures.
Eight research projects spanning the four domains were defined for the project-level evaluation. These projects embed models in an iterative discovery loop that is assessed with the accompanying sde-harness framework.
Question-level results
State-of-the-art models achieve lower scores on SDE than on general science benchmarks. Reported domain scores include 0.71 in biology (claude-opus-4.1), 0.60 in chemistry (claude-sonnet-4.5), 0.75 in materials (gpt-5), and 0.60 in physics (gpt-5). By comparison, models of the same class reach approximately 0.84 on MMMU-Pro and 0.86 on GPQA-Diamond.
Performance varies substantially across individual scenarios. For example, one leading model scored 0.85 on retrosynthesis planning but only 0.23 on NMR structure elucidation. Reasoning-enhanced models improve results relative to their base counterparts, yet further scaling of test-time compute and model size yields diminishing returns.
Top models from different providers show high correlation in their accuracy patterns (Spearman’s r > 0.8 in chemistry and physics) and frequently converge on the same incorrect answers for difficult items. On a hard subset of 86 questions, all evaluated models scored below 0.12.
Project-level results
On the eight discovery projects, models demonstrate measurable progress in several structured tasks, including protein sequence optimization, transition-metal complex optimization, and symbolic regression. In some cases, models locate optimal or near-optimal solutions within a limited number of iterations and explore Pareto fronts effectively.
The relationship between question-level and project-level performance is not strictly linear. Strong scenario scores often align with project success, yet exceptions occur: relatively weak performance on certain scenario questions does not always prevent effective navigation of the corresponding optimization project, while high accuracy on isolated retrosynthesis questions does not consistently translate into valid multi-step synthetic routes. No single model leads across all projects; relative ranking changes with the task.
Overall observations
The study reports a consistent performance gap between decontextualized science benchmarks and scenario-grounded discovery tasks, diminishing returns from further scaling of model size and reasoning effort, and systematic weaknesses shared across leading models from different providers. Large variation in scenario performance produces shifting rankings of the best model depending on the specific discovery project. One of our main questions is: Does AI increase or decrease trust in scientific discovery?
The authors note that current LLMs remain distant from general scientific “superintelligence.” At the same time, the models already show promise across a variety of scientific discovery projects, including cases in which constituent scenario scores are low. The SDE framework is presented as a reproducible benchmark intended to support further development of models for scientific discovery.
Data and code associated with the benchmark are available on public repositories.