Rafael Gorski

Trail log

Short and dated: what I'm testing, what I'm getting better at, and what I'd stop before it reaches production. These change more often than the rest of the site.

18 entries, the latest 17 October 2026, shown as Draft until confirmed · 3 long notes

Lab notes

Testing now6 entries

Trying it, not settled on it.

  1. Evals that start from production traces

    · Draft

    I'm treating evals the way test-driven development treats tests. Every failure I find in a production trace becomes a failing eval before anyone fixes it, and the suite gates each release module by module against a baseline. GN-0003 sets out the two waves and the loop between them.

  2. Tests that state what must never happen

    · Draft

    I'm trying property-based tests on the code around agents: refund limits, permissions, retry budgets. Instead of picking examples, I state what must never happen and let the framework try thousands of inputs to break it. What I'm watching for are failures that point to a rule nobody pinned down.

  3. A skill that makes the agent trace every feature to a problem

    · Draft

    I'm testing whether an agent skill can keep a coding agent honest about why it builds something. Problem-Based SRS walks it from customer problems to needs to requirements and checks that nothing is left without a parent. It's open source under MIT, for GitHub Copilot and Claude Code.

  4. Evals on every push, for prompts and skills

    · Draft

    Every push to the Problem-Based SRS repository runs plugin validation, unit tests, deterministic skill evals and a visual suite, then publishes a Skills Health dashboard. I want a prompt change that breaks a skill to fail the same way a code change that breaks the build does.

  5. Several coding agents, one site, separate branches

    · Draft

    This site was built by coding-agent sessions running in parallel: one curating sources, one on design directions, one on the radar, each on its own branch. What I'm watching is where the sessions drift apart, and how much of the work turns into review.

  6. One open trace format for agents

    · Draft

    I put the OpenTelemetry GenAI conventions in Trial, not Adopt. They name every part of an agent run, from workflow to tool call, but every document is still marked Development. GN-0001 sets out what I'd require now, including pinning the version so renames don't break dashboards.

Using more effectively5 entries

It works. I'm learning to use it better.

  1. The problem before the prompt

    · Draft

    Give a coding agent a feature request and it builds the feature. Give it the problem and who has it, and its output has something to be checked against. Without that, it fills the gap with plausible features. Naming the problem is the cheapest fix for agent output I know.

  2. Workflows before autonomous agents

    · Draft

    Most tasks I'd hand to an agent work better as a fixed workflow with one or two model calls. I reach for an open-ended agent loop only when the steps can't be known in advance, and then with a budget on calls.

  3. A baseline before the model

    · Draft

    Before any model work I want a simple baseline on real data: a rule, a lookup, a heuristic. If the model can't beat it, we stop early. If it can, we know by how much, and the failures get written down, not only the average.

  4. Cost per completed task, not per call

    · Draft

    I look at cost and at the number of model and tool calls per completed task. An agent that retries its way to an answer looks cheap per call and expensive per task. The GenAI conventions carry the token counts to do this per agent.

  5. Small branches and checks that run in seconds

    · Draft

    Coding agents produce more change than anyone can review carefully in one sitting. So keep their work on short branches, one concern each, behind checks that run in seconds. On this site that is a zero-dependency build and a link checker that refuses broken anchors.

Watch out for7 entries

I'd stop these before production. Each one is on Hold on the radar.

  1. Telemetry that lives only in a vendor's console

    · Draft

    If the only place to see what your agents did is one vendor's dashboard, you can't join it to the rest of your traces, and you can't leave. Put it in the contract: open conventions, exported to a backend you control.

  2. LLM judges nobody calibrated

    · Draft

    A model grading another model's output is cheap and fast, and it can be wrong in ways that look confident. Before a judge gates anything, check it against people grading the same samples, and check again when prompts or models change.

  3. Benchmark scores as launch evidence

    · Draft

    A leaderboard result says a model did well on someone else's tasks, once. It says little about your data, your tools or repeated runs. Launch on results from your own tasks, run several times, with the failures written down.

  4. Retries with no budget

    · Draft

    Automatic retries feel like resilience until a slow dependency turns every request into several, and the overload spreads. Retries need a budget, backoff and a stop, enforced by the platform rather than left to each agent.

  5. Capturing prompts and outputs by default

    · Draft

    Prompts, outputs and tool arguments often carry personal data. Capturing them makes debugging easier and turns your telemetry into a sensitive data store. Keep capture off unless a written policy turns it on for a named environment, with an owner and a retention period.

  6. A person approving every step as the only safeguard

    · Draft

    Approval prompts feel safe, but people stop reading them long before the agent stops asking. Keep approval for the decisions that need judgement, and enforce the rest with permissions, limits and checks outside the model.

  7. Blaming the model

    · Draft

    When an agent gets it wrong, "the model did it" is not an answer. A Canadian tribunal held an airline responsible for what its chatbot told a customer. Whoever deploys the system owns what it says and does, so name that owner before launch.

Notes

Longer notes, each pinned to the camp on the route where it matters.

  1. GN-0001 · Draft ·

    Agentic observability: what leaders should require before agents reach production

    Agents turn one request into a tree of decisions, model calls and tool calls. The OpenTelemetry GenAI conventions give that tree a shared vocabulary. Eight requirements to adopt.

  2. GN-0002 · Draft ·

    Test triangulation: properties that say what must never happen

    Example tests check only the cases an engineer thought of. Triangulate: state the properties that constrain the behaviour, including what must never happen, and let a framework try thousands of inputs to find a counterexample. Then map the value stream to see where late defects cost time.

  3. GN-0003 · Draft ·

    Eval-driven development: from production traces to the release gate

    Evals make an LLM system's limits explicit. Eval-driven development treats them as test-driven development treats tests: watch behaviour through telemetry, turn each failure into a failing eval, gate every change module by module against a baseline, and send what fails back to engineers.