Rafael Gorski

Gear check for the climb

The route shows when something matters. The radar shows what to carry: Adopt is what to take on the climb, Trial what to test on the approach, Assess what to scout ahead, and Hold what to leave behind. Each of the 40 blips is backed by sources in the field guide.

Ring
  1. Adopt Take on the climb. Use by default, because the evidence is consistent and the practice holds up at production scale.
  2. Trial Test on the approach. Use on a real workload with an owner and an exit plan; sound, but not yet settled.
  3. Assess Scout ahead. Study it and test it in isolation; it may matter, but evidence does not yet justify production use.
  4. Hold Leave behind. Avoid, and plan to retire where present; the evidence shows it fails, misleads or hides risk.
Mark
  • No change
  • New
  • Moved in toward Adopt
  • Moved out toward Hold

Edition October 2026 · draft · 17 adopt, 12 trial, 4 assess, 7 hold. Placements are editorial judgement; the marks will show how they move between editions.

The radar drawn as a disc: 40 numbered blips in four quadrants and four rings, from Adopt at the centre to Hold at the edge. Every blip is listed below with its ring, quadrant and evidence.

Techniques10 blips

Methods and practices

Adopt4 blips

  1. 1. Baseline before model

    Adopt · Techniques · New

    Measure a simple non-ML or single-call baseline first, and require any model or agent to beat it on the same task, metric and cost.

    On the route

  2. 2. Workflows before autonomous agents

    Adopt · Techniques · New

    Use the simplest design that works: a fixed workflow where steps are known, and open-ended agent control only where the task demonstrably needs it.

    On the route

  3. 3. Canary releases for model and prompt changes

    Adopt · Techniques · New

    Expose each model, prompt or configuration change to a small share of traffic against a control, with quality, latency and cost thresholds agreed before rollout.

    On the route

  4. 4. Blameless postmortems for AI incidents

    Adopt · Techniques · New

    Treat quality regressions, harmful outputs and agents acting out of scope as incidents, with written, blameless reviews that end in owned fixes and new evaluations.

    On the route

Trial4 blips

  1. 5. Problem-based requirements

    Trial · Techniques · New

    Derive requirements from named customer problems through customer needs, so every AI capability traces to a problem it solves and unneeded requirements are cut.

    On the route

  2. 6. Feedback-loop mapping

    Trial · Techniques · New

    Before scaling, draw the loops where a system's outputs return as its inputs, such as models trained on their own results, and name each delay and limit.

    On the route

  3. 7. Model cascades and routing

    Trial · Techniques · New

    Send each request to the cheapest model that meets the quality bar and escalate only when needed; one study matched the best single model at up to 98% lower cost.

    On the route

  4. 8. Injection-resistant agent design patterns

    Trial · Techniques · New

    Constrain what an agent can do after reading untrusted content, for example plan-then-execute. The patterns give provable resistance to prompt injection at an explicit cost in utility.

    On the route

Assess1 blip

  1. 9. Multi-agent decomposition

    Assess · Techniques · New

    Splitting work across cooperating agents adds documented failure modes in system design, inter-agent misalignment and task verification. Test against a single agent or workflow before committing.

    On the route

Hold1 blip

  1. 10. Unbudgeted automatic retries

    Hold · Techniques · New

    Agents and clients that retry without limits turn a slow dependency into overload. Cap retries with budgets and jittered backoff, and propagate deadlines through every step.

    On the route

Platforms and protocols7 blips

Shared infrastructure, interfaces and wire protocols

Adopt2 blips

  1. 11. W3C Trace Context across agent hops

    Adopt · Platforms and protocols · New

    Propagate one trace identifier through every agent, tool call, MCP server and queue, so a single request can be reconstructed end to end.

    On the route

  2. 12. OAuth to the current security BCP

    Adopt · Platforms and protocols · New

    Delegate agent access through OAuth as RFC 9700 recommends: audience-restricted, sender-constrained tokens. MCP authorization builds on OAuth, so the same baseline applies.

    On the route

Trial3 blips

  1. 13. OpenTelemetry GenAI semantic conventions

    Trial · Platforms and protocols · New

    The open vocabulary for model, agent and tool telemetry. Instrument now, but every document is still in Development status: pin and record the version each service emits.

    On the route

  2. 14. Model Context Protocol

    Trial · Platforms and protocols · New

    One open protocol for connecting AI applications to tools and data, now under Linux Foundation governance. Revisions still change: pin one and review its authorization section.

    On the route

  3. 15. Versioned open evaluation harness

    Trial · Platforms and protocols · New

    Keep evaluations as versioned code in an open harness, so the same suite reruns unchanged on every model, prompt or vendor change and results stay comparable.

    On the route

Assess1 blip

  1. 16. Agent2Agent protocol

    Assess · Platforms and protocols · New

    An open protocol for agents from different teams or vendors to discover each other and delegate tasks. Version 1.0 was released in March 2026; assess it where agents cross organisations.

    On the route

Hold1 blip

  1. 17. Telemetry only in a vendor's console

    Hold · Platforms and protocols · New

    Agent telemetry that cannot be exported in open formats leaves traces split by vendor and ties cost and quality reporting to one tool. GN-0001 requires open conventions instead.

    On the route

Measures11 blips

Metrics, evaluations and signals

Adopt4 blips

  1. 18. Continuous quality evaluation in production

    Adopt · Measures · New

    Offline evaluations missed production regressions in two public postmortems. Run quality evaluations continuously against live endpoints, and add each incident's failure as a new test.

    On the route

  2. 19. Cost reported with accuracy

    Adopt · Measures · New

    Report every evaluation result with its cost per task. Accuracy alone has led to needlessly complex, costly agents and to wrong conclusions about what drives gains.

    On the route

  3. 20. Tail latency per agent step

    Adopt · Measures · New

    Track p99 latency for each model and tool call and for the whole run. Rare slow calls come to dominate performance as agents chain more steps.

    On the route

  4. 21. Value-of-information measurement

    Adopt · Measures · New

    Treat a measurement as reducing uncertainty for a named decision. Use the value of information to decide what to measure, including benefits called intangible.

    On the route

Trial4 blips

  1. 22. Answer-quality SLOs

    Trial · Measures · New

    Give each AI feature a quality indicator, such as graded acceptability on sampled traffic, with an explicit objective beside latency and availability.

    On the route

  2. 23. Repeated-trial reliability (pass^k)

    Trial · Measures · New

    Run each task several times and report the chance that all attempts succeed. Single-run success overstates the consistency customers will experience.

    On the route

  3. 24. LLM judges calibrated to human grades

    Trial · Measures · New

    Strong models can approximate human preference ratings, but show position, verbosity and self-enhancement biases. Use them at scale only with a versioned human-graded sample to check agreement.

    On the route

  4. 25. Cost and calls per completed task

    Trial · Measures · New

    Attribute token cost to each agent and workflow, and watch the distribution of model and tool calls per invocation to catch runaway loops before the invoice does.

    On the route

Assess1 blip

  1. 26. Task time horizons

    Assess · Measures · New

    Rates models by the length of task, in skilled human time, they complete half the time. Useful for scoping delegation; the measure is new and tied to software tasks.

    On the route

Hold2 blips

  1. 27. LLM judges without human calibration

    Hold · Measures · New

    Grading criteria drift as people review outputs, and LLM judges carry documented biases, including self-enhancement. Without a human-graded check, scores cannot be trusted to track quality.

    On the route

  2. 28. Benchmark scores as launch evidence

    Hold · Measures · New

    Public agent benchmarks often have inadequate holdouts and ignore cost, and single runs hide inconsistency; models with equally strong test scores can behave differently in deployment.

    On the route

Governance12 blips

Standards, regulation and controls

Adopt7 blips

  1. 29. NIST AI RMF and Generative AI Profile

    Adopt · Governance · New

    A shared vocabulary for risk, legal and engineering: govern, map, measure and manage, with generative-AI risks such as confabulation. NIST has announced the framework is being revised.

    On the route

  2. 30. EU AI Act risk classification

    Adopt · Governance · New

    Where the Act applies, classify each system against its risk tiers now: duties fall on deployers as well as providers, and obligations are phasing in.

    On the route

  3. 31. Model cards and datasheets

    Adopt · Governance · New

    Keep a short, standard record for every deployed model and dataset: intended use, how it was evaluated, known gaps and uses that are out of scope.

    On the route

  4. 32. Per-agent identity and least privilege

    Adopt · Governance · New

    Give each agent its own identity, the narrowest permissions its task needs and logged activity, as for any insider with access to important systems.

    On the route

  5. 33. OWASP Top 10 launch review

    Adopt · Governance · New

    Map each LLM risk, from prompt injection to unbounded consumption, to a named control or an accepted risk before launch. The agentic list is newer and still settling.

    On the route

  6. 34. ISO/IEC 42001 AI management system

    Adopt · Governance · New

    A certifiable management-system standard for AI, published 2023. Map existing controls to it first; certify where customers or regulators need independent assurance.

    On the route

  7. 35. AI impact assessment before build

    Adopt · Governance · New

    Assess how a system may affect people and society before building it and again at each material change, and feed the results into the risk register.

    On the route

Trial1 blip

  1. 36. Controls enforced outside the model

    Trial · Governance · New

    Keep permissions, logging and a tested shutdown path in the harness around the model, not in its instructions, so an authorised person can always pause it mid-task.

    On the route

Assess1 blip

  1. 37. Shared AI incident disclosure

    Assess · Governance · New

    Disclose AI failures to those affected and share what failed, which controls broke and how to prevent it. The mechanisms for sharing across organisations are not yet established.

    On the route

Hold3 blips

  1. 38. Human approval as the only safeguard

    Hold · Governance · New

    Automation erodes the skills and attention people need to supervise it and take over when it fails. Pair human review with controls and evidence that do not depend on it.

    On the route

  2. 39. Shifting accountability to the model or provider

    Hold · Governance · New

    A tribunal held an airline liable for its chatbot's advice, and provider assurances do not transfer responsibility. Plan as though every AI output is the organisation's own statement.

    On the route

  3. 40. Capturing prompts and outputs by default

    Hold · Governance · New

    The OpenTelemetry conventions make content capture opt-in and warn it is likely to contain personal data. Enable it only under a written policy with an owner, retention and access controls.

    On the route

Read the same ground camp by camp on the route, or source by source in the field guide.