{"language": "en", "source_path": "README.md", "markdown": "# Scientific Evaluation Environments\n\n> **English** | [简体中文](./zh/README.md)\n\nIf you work on science, engineering, or AI, you usually hit the same problem fast: there are many papers about \"agent benchmarks\", but it is hard to tell what they actually measure, what kind of workflow they represent, and what you would learn from a failure.\n\nThis repository exists to make that legible.\n\nIt tracks work on evaluating scientific and engineering AI agents, but it does not stop at \"what score did the model get?\" The useful question is what the score helps you change next. If an agent fails to reproduce a paper, is the problem the plan, the tool use, the verifier, the task design, or the budget? Those are different failures, and they call for different repairs.\n\nIn other words, the repository follows this loop:\n\n![Evaluation loop](./site/assets/readme-eval-loop.svg)\n\nThe repository itself is simple:\n\n- One work card tells you what one paper or project actually contributed.\n- Three independent indexes help you find that same card from different questions.\n- Monthly reports explain what changed in the literature, not just which PDFs appeared.\n\n![Repository map](./site/assets/readme-repo-map.svg)\n\nThis is a reference library, not a benchmark implementation. The repository's [Explanation Style Guide](./EXPLANATION_STYLE.md) keeps the writing concrete: name the actor, show what changed, trace one real object when that helps, and say what the result still does not establish.\n\n---\n\n## Start here\n\nIf you already know what kind of question you have, use the matching entry point:\n\n| If your question is... | Start here |\n|---|---|\n| \"What evaluation problem is this paper really about?\" | [Topics](./topics/README.md) |\n| \"Which work exists for physics / chemistry / biology / robotics / civil engineering?\" | [Domains](./domains/README.md) |\n| \"Which benchmarks make agents do literature review, simulation, experiment design, or reproduction?\" | [Activities](./activities/README.md) |\n| \"What exactly does this one benchmark or method do?\" | [Works](./works/README.md) |\n| \"What was added recently, and why does it matter?\" | [Monthly reports](./monthly/README.md) |\n| \"What is the chronology by earliest public release?\" | [Works by first appearance](./WORKS_BY_DATE.md) |\n| \"I want the interactive view.\" | [SciEval explorer](https://yuema137.github.io/scieval/) |\n\nThese entry points are not competing taxonomies. They are three ways of reading the same corpus:\n\n```text\nTopic     → what evaluation problem does this work address?\nDomain    → which scientific or engineering field is it grounded in?\nActivity  → what does the evaluated agent actually do?\n```\n\nA single work can appear under several topics, several domains, and several activities at once.\n\n---\n\n## What is in scope\n\nThe short version is:\n\n- in scope when evaluation is the point\n- out of scope when evaluation is only the last results table\n\nMore concretely, this repository includes:\n\n- scientific and engineering agent benchmarks\n- evaluation methods for trajectories, judges, credit assignment, and resource use\n- work on benchmark validity, contamination, and verifier design\n- evaluation-driven improvement loops for skills, harnesses, data curation, and post-training\n- surveys and position papers that clarify the space\n\nIt does not include ordinary training or agent papers just because they report benchmark numbers. The cutline is operational: does evaluation define the objective, the feedback signal, the diagnosis, the intervention choice, or the experimental environment?\n\n## How to read the repository\n\nThe easiest way to avoid getting lost is to remember the division of labor:\n\n- [Works](./works/README.md) are factual cards for individual papers, benchmarks, methods, and surveys.\n- [Topics](./topics/README.md) are literature reviews about evaluation questions.\n- [Domains](./domains/README.md) tell you what scientific or engineering field a work is grounded in.\n- [Activities](./activities/README.md) tell you what the agent actually does.\n- [Monthly reports](./monthly/README.md) explain what changed month by month.\n\nThe Markdown files are the ground truth. The HTML explorer is only a render layer generated from those files.\n\n## Current coverage\n\nThe repository currently contains:\n\n- **382** work cards\n- **15** topic pages\n- **19** domain pages\n- **11** activity pages\n- **32** monthly reports from 2024-01 through 2026-08\n\nCoverage is broad across both sciences and engineering. The heaviest current concentrations are in:\n\n- scientific agent benchmarks\n- trajectory evaluation\n- physics, chemistry, biology, astronomy, and civil/structural engineering\n- scientific problem solving, scientific software/workflow engineering, and data analysis\n\n## Browse by Topic\n\nA topic starts with an evaluation problem. Some topics ask what behavior to measure, such as planning or long-horizon work. Others ask whether the measurement itself is trustworthy, or how its feedback changes skills, harnesses, data, and post-training. Each topic page explains that problem, groups the main approaches, and compares them using dimensions that fit the problem. See [`topics/`](./topics/README.md) for the full index.\n\n| Topic | What you'll find |\n|---|---|\n| [General Long-Horizon Agent Benchmarks](./topics/long_horizon_evaluation.md) | Benchmarks whose tasks need many sequential decisions, tool calls, or turns — where failures accumulate and intermediate state matters. |\n| [Scientific Agent Benchmarks](./topics/scientific_agents.md) | Agents on tasks drawn from real scientific research and practice, judged against published or expert-defined outcomes. |\n| [Planning & Decision-Making Evaluation](./topics/planning_decision_evaluation.md) | Whether an agent selects a sound plan or next action from the current state, goals, constraints, tools, and evidence, and replans appropriately after feedback. |\n| [Hierarchical Decision Abstraction](./topics/hierarchical_decision_abstraction.md) | How agent behavior should be represented, evaluated, and optimized across goals, strategies, subgoals, semantic actions, primitive actions, and control signals. |\n| [Trajectory Evaluation](./topics/trajectory_evaluation.md) | Methods that score the whole sequence of actions and intermediate states, not just the final answer. |\n| [Skill Hierarchy](./topics/skill_hierarchy.md) | Decomposing a complex capability into narrower subskills, each scored separately. |\n| [Credit Assignment](./topics/credit_assignment.md) | Attributing a trajectory's success or failure to specific steps or subgoals — dense rewards, partial credit, per-step scoring. |\n| [Resource-aware Evaluation](./topics/resource_aware_evaluation.md) | Treating tokens, fees, wall-clock time, or compute as part of what the benchmark measures — sometimes as an explicit objective. |\n| [Evaluator Reliability & Validation](./topics/evaluator_reliability_validation.md) | Validating judges, reward models, rubrics, and verifiers against human or deterministic ground truth and downstream use. |\n| [Benchmark Design, Validity & Contamination](./topics/benchmark_design_validity_contamination.md) | Task construction, verifier rigor, contamination resistance, dynamic evaluation, and ecological validity. |\n| [Skill Learning & Evolution](./topics/skill_learning_evolution.md) | Turning experience and evaluation feedback into reusable skills, then testing transfer and failure modes. |\n| [Agent Harnesses & Scaffolding](./topics/agent_harnesses_scaffolding.md) | Measuring, attributing, and optimizing the control structures surrounding a model. |\n| [Evaluation-Driven Data Curation](./topics/evaluation_driven_data_curation.md) | Revising selection, generation, filtering, or mixture policies from downstream evaluation feedback. |\n| [Evaluation-Driven Post-Training](./topics/evaluation_driven_post_training.md) | Using evaluation as an objective, feedback signal, or experimental environment for model and agent improvement. |\n| [Survey](./topics/survey.md) | Surveys and position papers on agent evaluation — an index of references rather than a task suite. |\n\n---\n\n## Browse by Domain\n\nDomain pages answer a narrower question: where does the evaluated work happen? A physics benchmark and a biology benchmark may use the same evaluation method but face different tools, artifacts, costs, and correctness standards. The counts below show current coverage; the maintained index and per-domain tables live in [`domains/`](./domains/README.md).\n\n**Sciences**\n\n| Domain | Works |\n|---|--:|\n| [Physics](./domains/physics.md) | 48 |\n| [Chemistry](./domains/chemistry.md) | 39 |\n| [Biology](./domains/biology.md) | 38 |\n| [Materials Science](./domains/materials_science.md) | 28 |\n| [AI & Machine Learning Research](./domains/ai_ml_research.md) | 27 |\n| [Mathematics](./domains/mathematics.md) | 19 |\n| [Medicine & Health](./domains/medicine_health.md) | 22 |\n| [Neuroscience & Cognitive Science](./domains/neuroscience_cognitive_science.md) | 13 |\n| [Astronomy](./domains/astronomy.md) | 34 |\n| [Earth Science](./domains/earth_science.md) | 12 |\n| [Computer Science](./domains/computer_science.md) | 7 |\n| [Environmental Science](./domains/environmental_science.md) | 6 |\n\n**Engineering**\n\n| Domain | Works |\n|---|--:|\n| [Electrical Engineering](./domains/electrical_engineering.md) | 18 |\n| [Robotics](./domains/robotics.md) | 18 |\n| [Software & Systems Engineering](./domains/software_systems_engineering.md) | 18 |\n| [Mechanical & Aerospace Engineering](./domains/mechanical_aerospace_engineering.md) | 12 |\n| [Energy Systems](./domains/energy_systems.md) | 5 |\n| [Civil & Structural Engineering](./domains/civil_structural_engineering.md) | 30 |\n| [Chemical Engineering](./domains/chemical_engineering.md) | 12 |\n\nNarrower fields fold into these canonical domains (bioinformatics → Biology, GIS → Earth Science, psychology → Neuroscience & Cognitive Science, …), and a work may appear in several domains. Web/UI agents, computer use, and pure evaluation methodology are not science or engineering domains and do not appear here.\n\n---\n\n## Browse by Research Activity\n\nActivity pages follow the work itself. Does the agent search literature, run a simulation, design an experiment, reproduce a result, or carry a project end to end? The same activity can appear in several domains and use several evaluation methods. Work that evaluates no scientific or research task, such as a survey or pure evaluation methodology, carries an explicit `N/A`. See [`activities/`](./activities/README.md) for the full index.\n\n| Activity | What it covers | Works |\n|---|---|--:|\n| [Scientific Problem Solving & Reasoning](./activities/scientific_problem_solving_reasoning.md) | Scientific QA, derivations, proofs, quantitative and multimodal problem solving, diagnostic reasoning | 94 |\n| [Scientific Software & Workflow Engineering](./activities/scientific_software_workflow_engineering.md) | Scientific/engineering code, repository and pipeline engineering, HDL and formal-spec code | 71 |\n| [Data Analysis & Statistical Inference](./activities/data_analysis_statistical_inference.md) | Statistical analysis and inference, bioinformatics/omics analysis, data interpretation | 44 |\n| [Experiment Design & Scientific Discovery](./activities/experiment_design_discovery.md) | Experiment and observation planning, hypothesis generation, law discovery | 23 |\n| [Simulation & Scientific Computing](./activities/simulation_scientific_computing.md) | Numerical simulation, PDE/FEM, MD/DFT, running and building scientific simulators | 35 |\n| [Modeling & Prediction](./activities/modeling_prediction.md) | Predictive and surrogate modelling, property prediction, forecasting | 22 |\n| [Optimization & Engineering Design](./activities/optimization_engineering_design.md) | Parameter and controller tuning, engineering/inverse design, materials and molecular design | 26 |\n| [Literature Search & Evidence Synthesis](./activities/literature_evidence_synthesis.md) | Literature retrieval, systematic review, evidence synthesis, literature-grounded extraction | 23 |\n| [Research Reproduction & Replication](./activities/research_reproduction_replication.md) | Reproducing published analyses, results, and methods; matching reported findings | 11 |\n| [End-to-End Research](./activities/end_to_end_research.md) | Multi-stage research lifecycle across several major phases | 9 |\n| [Laboratory & Instrument Control](./activities/laboratory_instrument_control.md) | Instrument, microscope, and beamline control; lab automation; behaviour-defined control code | 3 |\n\n---\n\n## Scope\n\nWork is in scope when evaluation changes what we know or what the development loop does next. That includes benchmarks, diagnostic methods, evaluator validation, benchmark-validity research, scientific workflows, and systems that use evaluation to revise skills, harnesses, data, or post-training.\n\nA paper is not in scope merely because it reports benchmark scores. Pure training, optimization, data, memory, or multi-agent work stays out when evaluation appears only in the final results table. The test is operational: does evaluation define the objective, supply feedback, select an intervention, diagnose a failure, or serve as the experiment environment?\n\n\"Works\" is broader than \"benchmarks\": the collection holds cards for benchmarks, evaluation methodologies, evaluation frameworks, evaluation-focused RL contributions, surveys, and position papers. Each card notes its type explicitly. The collection currently holds **382 work cards**, **15 topic pages**, **19 domain pages**, and **11 activity pages**, each mirrored in Chinese under [`zh/`](./zh/README.md).\n\n---\n\n## A Living Knowledge Base\n\nThis repository is maintained continuously.\n\nEvery three days, an update agent searches public sources for new work, drafts cards and index changes, and opens a pull request for human review. On the first day of each month, the repository also prepares a bilingual monthly report for the previous month. Both the daily update path and the monthly report path support manual triggering as well, so maintainers can refresh a specific window without inventing duplicate files.\n\n## Interactive explorer\n\nThe repository is good for maintenance and provenance. It is less good for visual browsing. The companion explorer fixes that without changing the source of truth.\n\nThe explorer:\n\n- is deployed as an isolated `/scieval/` subpage on the maintainer's GitHub Pages site\n- reads generated JSON exported from repository Markdown\n- shows the corpus as a searchable, filterable interface\n- adds visual structure such as the first-appearance timeline and axis overview\n\nAfter relevant Markdown changes merge here, an automated workflow builds and\nvalidates a self-contained snapshot. A guarded pull request then updates only\n`scieval/**` in the personal-site repository. If that process fails, the\nprevious snapshot and the rest of the personal site stay unchanged.\n\nThe explorer does **not** own content. If the site and a Markdown file disagree, the Markdown file wins.\n\n---\n\n## Repository Structure\n\nThe knowledge base has **four layers**: the works layer, plus three co-equal aggregation axes over it.\n\n| Directory | Role |\n|---|---|\n| [`works/`](./works/README.md) | One factual reference card per work. Flat, kebab-case, one Markdown file each. |\n| [`topics/`](./topics/README.md) | Literature-review pages — the evaluation-research axis spanning measurement, diagnosis, and improvement. Each topic owns its own comparison table. |\n| [`domains/`](./domains/README.md) | Field-axis reference pages, one per canonical science or engineering domain, with a fixed-column comparison table. |\n| [`activities/`](./activities/README.md) | Task-axis reference pages, one per canonical research activity, with Definition, Scope, task patterns, and a comparison table. |\n| [`zh/`](./zh/README.md) | Chinese mirror of every page, synced after each English batch. |\n\nTwo navigational conventions keep the axes in sync: each card's `Topics` block links up to its topics, and each topic page's `Related Works` links back down to its cards. The domain mapping is maintained one-way on the domain pages. Root-level [`AGENT.md`](./AGENT.md) is the repository constitution and [`CLAUDE.md`](./CLAUDE.md) is its quick reference; each directory's own `README.md` documents its page template and rules.\n\n---\n\n## Contributing\n\nContributions are welcome. Automated updates complement, rather than replace, community contributions; missing or newly relevant work can still be proposed manually. All contributor and maintainer rules — reference validation, page templates, the canonical taxonomies, and the bilingual sync cadence — live in [`AGENT.md`](./AGENT.md) (the constitution) and [`CLAUDE.md`](./CLAUDE.md) (its quick reference), with layer-specific rules in each directory's README. Every page is available in English and Chinese; use the language switcher at the top of any page.\n"}