A working guide to how self-driving systems are checked, trusted and measured — and, more generally, to evaluating any system whose failures are rare, expensive and safety-critical.
The document is organised as two rounds. Round 01 — V&V asks what it takes to trust a simulator: how verification differs from validation, how evidence is tiered from unit tests up to closed-course driving, and how a real programme spends its compute and triages its failures. Round 02 — Metrics asks how to design a measure that supports an actual decision, and works through the four places measurement usually goes wrong: Goodhart effects, stratification, sparse events, and uncertainty.
I first wrote this to prepare for a validation-and-verification interview, so it is shaped around the questions that come up in those conversations — mock cases at both module and architecture level, a searchable glossary, and a one-page summary for the ten minutes before you walk in. It has since turned out to work reasonably well as a standalone introduction to AV evaluation, which is why it is here.
一份关于自动驾驶系统如何被检验、被信任、被度量的工作笔记——更一般地说，也是关于如何评估任何一个「失效罕见、代价高昂、且性命攸关」的系统。
文档分为两轮。第一轮 · V&V 讨论「凭什么相信一个仿真器」：verification 与 validation 的区别、证据如何从单元测试一路分层到闭场路测、以及真实项目怎么分配算力、怎么 triage 失败。第二轮 · Metrics 讨论如何设计一个真正支撑决策的指标，并逐一走过度量最容易翻车的四个地方：Goodhart 效应、分层、稀疏事件、以及不确定度。
这份材料最初是我为一场 validation and verification 面试准备的，所以它是围绕那类对话中会出现的问题组织的——模块级和架构级的 mock case、可搜索的术语表、以及一页纸的考前速览。后来发现它本身也算一份还不错的 AV 评估入门，所以放在这里。
These generalise well beyond driving. Most of what is here applies to any agent or system evaluated on outcomes that are costly to observe — which is, more or less, what I work on.
这些结论远不止适用于自动驾驶。这里的大部分内容，对任何「结果观测代价高昂」的 agent 或系统评估都成立——这大致也就是我在做的事。