Search
Search the notes, the trail log, the radar and the field guide. A theme reads as its whole family of words: evals also finds evaluation, benchmark and judge.
Themes
The site indexed by theme, as at the back of an atlas. Each line is filed where it is written; a note lists the sections that carry the theme.
-
Requirements
3 notes · 1 lab note · 1 blip · 6 sources
- GN-0001 Agentic observability, original note
Section 2: The shared vocabulary now existsSection 6: Requirements for leadershipSection 8: What is still open
- GN-0002 Test triangulation, original note
Section 1: One angle is not enoughSection 2: Four angles on one behaviourSection 6: A counterexample is a question about the constraint setSection 7: Map the value stream to find the wasteSection 8: Requirements for leadership
- GN-0003 Eval-driven development, original note
Section 1: Evals make the limits explicitSection 2: Eval-driven development, step by stepSection 6: Requirements for leadership
- In the trail log: A skill that makes the agent trace every feature to a problem
- On the radar: Problem-based requirements · Trial
- In the field guide: Problem-Based Software Requirements Specification · 2016
- In the field guide: Building effective agents · 2024
- In the field guide: Effective Software Testing · 2022
- In the field guide: Model Context Protocol Specification · 2026
- In the field guide: Agent2Agent (A2A) Protocol Specification · 2026
- In the field guide: Semantic conventions for generative AI systems · living
- GN-0001 Agentic observability, original note
-
Testing
2 notes · 1 position · 4 lab notes · 4 blips · 8 sources
- GN-0002 Test triangulation, original note
Section 1: One angle is not enoughSection 2: Four angles on one behaviourSection 3: Properties, not examplesSection 4: Write down what must never happenSection 5: Thousands of cases, one counterexampleSection 6: A counterexample is a question about the constraint setSection 7: Map the value stream to find the wasteSection 8: Requirements for leadershipSection 9: Where to startSection 10: What is still open
- GN-0003 Eval-driven development, original note
Section 1: Evals make the limits explicitSection 2: Eval-driven development, step by stepSection 4: Wave 2: a guardrail for every module in CI/CDSection 5: Close the loop: from eval results back to engineersSection 6: Requirements for leadershipSection 8: What is still open
- On the home page: Evaluations are the spec you can run.
- In the trail log: Evals that start from production traces
- In the trail log: Tests that state what must never happen
- In the trail log: A skill that makes the agent trace every feature to a problem
- In the trail log: Evals on every push, for prompts and skills
- On the radar: Multi-agent decomposition · Assess
- On the radar: Continuous quality evaluation in production · Adopt
- On the radar: Benchmark scores as launch evidence · Hold
- On the radar: Controls enforced outside the model · Trial
- In the field guide: Underspecification Presents Challenges for Credibility in Modern… · 2022
- In the field guide: Effective Software Testing · 2022
- In the field guide: Reliable Product Launches at Scale · 2016
- In the field guide: The ML Test Score · 2017
- In the field guide: Expanding on what we missed with sycophancy · 2025
- In the field guide: Addressing Cascading Failures · 2016
- In the field guide: FrugalGPT · 2023
- In the field guide: Practices for Governing Agentic AI Systems · 2023
- GN-0002 Test triangulation, original note
-
Evals
3 notes · 1 position · 5 lab notes · 9 blips · 12 sources
- GN-0001 Agentic observability, original note
Section 2: The shared vocabulary now existsSection 4: From executive questions to signalsSection 6: Requirements for leadershipSection 7: A first ninety daysSection 8: What is still open
- GN-0002 Test triangulation, original note
- GN-0003 Eval-driven development, original note
Section 1: Evals make the limits explicitSection 2: Eval-driven development, step by stepSection 3: Wave 1: watch behaviour through telemetrySection 4: Wave 2: a guardrail for every module in CI/CDSection 5: Close the loop: from eval results back to engineersSection 6: Requirements for leadershipSection 7: Where to startSection 8: What is still open
- On the home page: Evaluations are the spec you can run.
- In the trail log: Evals that start from production traces
- In the trail log: Evals on every push, for prompts and skills
- In the trail log: LLM judges nobody calibrated
- In the trail log: Benchmark scores as launch evidence
- In the trail log: A person approving every step as the only safeguard
- On the radar: Blameless postmortems for AI incidents · Adopt
- On the radar: Versioned open evaluation harness · Trial
- On the radar: Continuous quality evaluation in production · Adopt
- On the radar: Cost reported with accuracy · Adopt
- On the radar: LLM judges calibrated to human grades · Trial
- On the radar: LLM judges without human calibration · Hold
- On the radar: Benchmark scores as launch evidence · Hold
- On the radar: Model cards and datasheets · Adopt
- On the radar: Controls enforced outside the model · Trial
- In the field guide: τ-bench · 2024
- In the field guide: AI Agents That Matter · 2024
- In the field guide: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena · 2023
- In the field guide: Who Validates the Validators? · 2024
- In the field guide: Underspecification Presents Challenges for Credibility in Modern… · 2022
- In the field guide: Inspect · living
- In the field guide: Datasheets for Datasets · 2021
- In the field guide: Reliable Product Launches at Scale · 2016
- In the field guide: Expanding on what we missed with sycophancy · 2025
- In the field guide: Evals for AI Engineers · 2026
- In the field guide: A postmortem of three recent issues · 2025
- In the field guide: Model Cards for Model Reporting · 2019
- GN-0001 Agentic observability, original note
-
Rollout
3 notes · 2 lab notes · 5 blips · 11 sources
- GN-0001 Agentic observability, original note
- GN-0002 Test triangulation, original note
Section 7: Map the value stream to find the wasteSection 8: Requirements for leadershipSection 9: Where to start
- GN-0003 Eval-driven development, original note
Section 1: Evals make the limits explicitSection 3: Wave 1: watch behaviour through telemetrySection 4: Wave 2: a guardrail for every module in CI/CDSection 5: Close the loop: from eval results back to engineersSection 6: Requirements for leadershipSection 7: Where to startSection 8: What is still open
- In the trail log: Evals that start from production traces
- In the trail log: Blaming the model
- On the radar: Canary releases for model and prompt changes · Adopt
- On the radar: Agent2Agent protocol · Assess
- On the radar: Benchmark scores as launch evidence · Hold
- On the radar: EU AI Act risk classification · Adopt
- On the radar: Model cards and datasheets · Adopt
- In the field guide: Underspecification Presents Challenges for Credibility in Modern… · 2022
- In the field guide: Reliable Product Launches at Scale · 2016
- In the field guide: Canarying Releases · 2018
- In the field guide: Guidelines for secure AI system development · 2023
- In the field guide: Expanding on what we missed with sycophancy · 2025
- In the field guide: Evals for AI Engineers · 2026
- In the field guide: Best Current Practice for OAuth 2.0 Security · 2025
- In the field guide: Lean Software Development · 2003
- In the field guide: EU Artificial Intelligence Act · 2024
- In the field guide: Models as Insider Risks in the Super Intelligence Era · 2026
- In the field guide: Model Cards for Model Reporting · 2019
-
Tools and protocols
2 notes · 1 position · 2 lab notes · 6 blips · 6 sources
- GN-0001 Agentic observability, original note
Section 1: Agents are distributed systems that make decisionsSection 2: The shared vocabulary now existsSection 3: What one agent run looks likeSection 4: From executive questions to signalsSection 6: Requirements for leadershipSection 7: A first ninety days
- GN-0003 Eval-driven development, original note
- On the home page: If I can't see what an agent did, I can't own it.
- In the trail log: One open trace format for agents
- In the trail log: Cost per completed task, not per call
- On the radar: W3C Trace Context across agent hops · Adopt
- On the radar: OAuth to the current security BCP · Adopt
- On the radar: Model Context Protocol · Trial
- On the radar: Agent2Agent protocol · Assess
- On the radar: Tail latency per agent step · Adopt
- On the radar: Cost and calls per completed task · Trial
- In the field guide: Model Context Protocol Specification · 2026
- In the field guide: Agent2Agent (A2A) Protocol Specification · 2026
- In the field guide: The Tail at Scale · 2013
- In the field guide: Best Current Practice for OAuth 2.0 Security · 2025
- In the field guide: Semantic conventions for generative AI systems · living
- In the field guide: Trace Context · 2021
- GN-0001 Agentic observability, original note
-
Cost
3 notes · 3 positions · 4 lab notes · 9 blips · 9 sources · 1 stage
- GN-0001 Agentic observability, original note
Section 1: Agents are distributed systems that make decisionsSection 3: What one agent run looks likeSection 4: From executive questions to signalsSection 6: Requirements for leadershipSection 7: A first ninety daysSection 8: What is still open
- GN-0002 Test triangulation, original note
Section 4: Write down what must never happenSection 6: A counterexample is a question about the constraint setSection 7: Map the value stream to find the wasteSection 9: Where to start
- GN-0003 Eval-driven development, original note
Section 4: Wave 2: a guardrail for every module in CI/CDSection 8: What is still open
- On the home page: Start from the problem, not the model.
- On the home page: If I can't see what an agent did, I can't own it.
- On the home page: Put the controls outside the model.
- In the trail log: Tests that state what must never happen
- In the trail log: Workflows before autonomous agents
- In the trail log: Cost per completed task, not per call
- In the trail log: Retries with no budget
- On the radar: Baseline before model · Adopt
- On the radar: Canary releases for model and prompt changes · Adopt
- On the radar: Model cascades and routing · Trial
- On the radar: Injection-resistant agent design patterns · Trial
- On the radar: Unbudgeted automatic retries · Hold
- On the radar: Telemetry only in a vendor's console · Hold
- On the radar: Cost reported with accuracy · Adopt
- On the radar: Cost and calls per completed task · Trial
- On the radar: Benchmark scores as launch evidence · Hold
- In the field guide: Rules of Machine Learning · living
- In the field guide: Hidden Technical Debt in Machine Learning Systems · 2015
- In the field guide: AI Agents That Matter · 2024
- In the field guide: Reliable Product Launches at Scale · 2016
- In the field guide: Canarying Releases · 2018
- In the field guide: Addressing Cascading Failures · 2016
- In the field guide: FrugalGPT · 2023
- In the field guide: FinOps for AI Overview · living
- In the field guide: Dapper, a Large-Scale Distributed Systems Tracing Infrastructure · 2010
- On the route: Scale · Can it grow in users, agents and cost without losing control?
- GN-0001 Agentic observability, original note
-
Security
2 notes · 4 blips · 9 sources
- GN-0001 Agentic observability, original note
- GN-0002 Test triangulation, original note
- On the radar: Injection-resistant agent design patterns · Trial
- On the radar: OAuth to the current security BCP · Adopt
- On the radar: Per-agent identity and least privilege · Adopt
- On the radar: OWASP Top 10 launch review · Adopt
- In the field guide: Inspect · living
- In the field guide: OWASP Top 10 for LLM Applications 2025 · 2024
- In the field guide: OWASP Top 10 for Agentic Applications for 2026 · 2025
- In the field guide: Guidelines for secure AI system development · 2023
- In the field guide: Adversarial Machine Learning · 2025
- In the field guide: Model Context Protocol Specification · 2026
- In the field guide: Agent2Agent (A2A) Protocol Specification · 2026
- In the field guide: Design Patterns for Securing LLM Agents against Prompt Injections · 2025
- In the field guide: Best Current Practice for OAuth 2.0 Security · 2025
-
Observability
3 notes · 2 positions · 6 lab notes · 6 blips · 8 sources
- GN-0001 Agentic observability, original note
Section 1: Agents are distributed systems that make decisionsSection 2: The shared vocabulary now existsSection 3: What one agent run looks likeSection 4: From executive questions to signalsSection 5: Content capture is a governance decisionSection 6: Requirements for leadershipSection 7: A first ninety daysSection 8: What is still open
- GN-0002 Test triangulation, original note
Section 6: A counterexample is a question about the constraint set
- GN-0003 Eval-driven development, original note
Section 2: Eval-driven development, step by stepSection 3: Wave 1: watch behaviour through telemetrySection 5: Close the loop: from eval results back to engineersSection 6: Requirements for leadershipSection 7: Where to startSection 8: What is still open
- On the home page: Start from the problem, not the model.
- On the home page: If I can't see what an agent did, I can't own it.
- In the trail log: Evals that start from production traces
- In the trail log: A skill that makes the agent trace every feature to a problem
- In the trail log: Evals on every push, for prompts and skills
- In the trail log: One open trace format for agents
- In the trail log: Telemetry that lives only in a vendor's console
- In the trail log: Capturing prompts and outputs by default
- On the radar: Problem-based requirements · Trial
- On the radar: W3C Trace Context across agent hops · Adopt
- On the radar: OpenTelemetry GenAI semantic conventions · Trial
- On the radar: Telemetry only in a vendor's console · Hold
- On the radar: Controls enforced outside the model · Trial
- On the radar: Capturing prompts and outputs by default · Hold
- In the field guide: Problem-Based Software Requirements Specification · 2016
- In the field guide: The ML Test Score · 2017
- In the field guide: Evals for AI Engineers · 2026
- In the field guide: Why Do Multi-Agent LLM Systems Fail? · 2025
- In the field guide: Semantic conventions for generative AI systems · living
- In the field guide: Trace Context · 2021
- In the field guide: Dapper, a Large-Scale Distributed Systems Tracing Infrastructure · 2010
- In the field guide: EU Artificial Intelligence Act · 2024
- GN-0001 Agentic observability, original note
-
Reliability
3 notes · 5 blips · 10 sources
- GN-0001 Agentic observability, original note
Section 3: What one agent run looks likeSection 6: Requirements for leadership
- GN-0002 Test triangulation, original note
- GN-0003 Eval-driven development, original note
Section 3: Wave 1: watch behaviour through telemetrySection 4: Wave 2: a guardrail for every module in CI/CDSection 5: Close the loop: from eval results back to engineers
- On the radar: Blameless postmortems for AI incidents · Adopt
- On the radar: Continuous quality evaluation in production · Adopt
- On the radar: Answer-quality SLOs · Trial
- On the radar: Repeated-trial reliability (pass^k) · Trial
- On the radar: Shared AI incident disclosure · Assess
- In the field guide: Reliable Product Launches at Scale · 2016
- In the field guide: Canarying Releases · 2018
- In the field guide: Guidelines for secure AI system development · 2023
- In the field guide: Expanding on what we missed with sycophancy · 2025
- In the field guide: Addressing Cascading Failures · 2016
- In the field guide: Service Level Objectives · 2016
- In the field guide: Postmortem Culture · 2016
- In the field guide: A postmortem of three recent issues · 2025
- In the field guide: Model Cards for Model Reporting · 2019
- In the field guide: Moffatt v. Air Canada, 2024 BCCRT 149 · 2024
- GN-0001 Agentic observability, original note
-
Human oversight
1 note · 1 position · 1 lab note · 2 blips · 3 sources
- GN-0003 Eval-driven development, original note
- On the home page: Put the controls outside the model.
- In the trail log: A person approving every step as the only safeguard
- On the radar: Model cascades and routing · Trial
- On the radar: Human approval as the only safeguard · Hold
- In the field guide: EU Artificial Intelligence Act · 2024
- In the field guide: Practices for Governing Agentic AI Systems · 2023
- In the field guide: Ironies of automation · 1983
-
Governance
3 notes · 1 position · 1 lab note · 6 blips · 10 sources · 1 stage
- GN-0001 Agentic observability, original note
Section 1: Agents are distributed systems that make decisionsSection 3: What one agent run looks likeSection 5: Content capture is a governance decisionSection 6: Requirements for leadershipSection 8: What is still open
- GN-0002 Test triangulation, original note
- GN-0003 Eval-driven development, original note
Section 1: Evals make the limits explicitSection 4: Wave 2: a guardrail for every module in CI/CDSection 5: Close the loop: from eval results back to engineersSection 6: Requirements for leadershipSection 7: Where to start
- On the home page: Put the controls outside the model.
- In the trail log: Capturing prompts and outputs by default
- On the radar: Model Context Protocol · Trial
- On the radar: NIST AI RMF and Generative AI Profile · Adopt
- On the radar: EU AI Act risk classification · Adopt
- On the radar: ISO/IEC 42001 AI management system · Adopt
- On the radar: Shifting accountability to the model or provider · Hold
- On the radar: Capturing prompts and outputs by default · Hold
- In the field guide: Adversarial Machine Learning · 2025
- In the field guide: Dapper, a Large-Scale Distributed Systems Tracing Infrastructure · 2010
- In the field guide: Postmortem Culture · 2016
- In the field guide: EU Artificial Intelligence Act · 2024
- In the field guide: ISO/IEC 42001 AI management system · 2023
- In the field guide: ISO/IEC 42005 AI system impact assessment · 2025
- In the field guide: AI RMF 1.0 · 2023
- In the field guide: AI RMF Generative AI Profile · 2024
- In the field guide: Practices for Governing Agentic AI Systems · 2023
- In the field guide: Moffatt v. Air Canada, 2024 BCCRT 149 · 2024
- On the route: Govern · Who is accountable, and can we prove it?
- GN-0001 Agentic observability, original note