I had a pipeline with a drafting step feeding a verification step, and I upgraded the drafting model to a newer, better one. Hallucinated citations in the finished output went up.

The newer model was not worse. It was more fluent. Its prose carried more conviction and the URLs it invented looked more like real URLs — so the verifier downstream, which decided how hard to check based on how the draft read, checked less often. A real improvement to one stage made the system worse, and the regression existed nowhere except in the pair. Evaluate either stage on its own and both look fine.

I cannot show you that data. It predates the tracking and it is gone, which is most of the reason the tracking exists.

The library itself came out of something duller. Several AI-assisted projects here each fanned parallel LLM calls at external providers, and each had been built differently — prompts, config and clients wherever the assistant happened to drop them. Nobody could say what a batch had cost, which prompt text produced a given answer, or why last night’s run stopped halfway.

What it does

See what last night's batch cost, per model and agent

Open /cost-by-model and /cost-by-agent and you can see which step of a twelve-step pipeline burned the money — then drill into a single call and read the exact prompt text that produced the answer, at the version it was sent. Every page has a matching /api/*.json if you would rather query it than look at it.

Stop a looping prompt before it eats the month

check_budget() runs before the request leaves the process and raises instead of sending it. Caps are daily or monthly, scoped to an agent, a job kind or a tag. One trap the docs name: a Claude Max backend reports $0 on every call, so a dollar cap never trips there — cap tokens or calls. BUDGET_ENFORCEMENT_DISABLED=true turns the guard off mid-incident.

Catch the run that quietly dropped half its rows

A model that hits max_tokens mid-array leaves valid JSON with the tail gone. Recover it naively and you get 40 records in, 15 rows out, exit 0. parse_llm_json(..., on_truncation="raise") fails the run instead, and tracked_call defaults to that. Runs the provider reported as cut off are tagged truncated, so one query finds every one in a batch.

Restart a crashed batch without redoing finished work

pf-jobs list --status failed finds them, pf-jobs show <id> gives the step history and event timeline, pf-jobs retry <id> picks the job back up. Steps that already succeeded short-circuit rather than run twice. Calls made inside a job record its id on their own, so cost stays attributed with no job id threaded through every signature.

Run a batch on Claude Max instead of API credits

OpenRouter, the Anthropic SDK and the local claude --print CLI all answer the same chat(messages, model) call. A YAML file says which one each agent uses, and an edit takes effect within 60 seconds without a restart. Claude Max calls bill nothing and still land in the same tables with real token counts. Pin a model when you batch, or the CLI uses whatever your interactive session is on.

Test a new model against work you already approved

Tag runs you reviewed into a golden set. EvalRunner replays their stored prompts against the new model or prompt version and scores each result field by field, or with a judge model. No console script ships — you write the short runner that exits 1 when report.passed is false, and CI gates the merge on that.

What that looks like in practice

Here is that same question, asked against a database that did keep the records.

Logs will not answer it. Logs record what happened; they do not record what each call was — which stage, which model, which prompt version, which pipeline it belonged to. Without those on one row the question is not hard, it is unaskable.

-- Does the model used at one stage predict how the NEXT stage behaves?
-- Runs are linked by the pipeline they belong to, so the stages can be joined.
WITH upstream AS (
  SELECT r.draft_id, m.name AS upstream_model
  FROM   llm_runs r
  JOIN   llm_agent_types a ON a.id = r.agent_type_id
  JOIN   llm_models      m ON m.id = r.model_id
  WHERE  a.slug = 'matcher' AND r.draft_id IS NOT NULL
  GROUP  BY r.draft_id, m.name
),
uncontested AS (                    -- drafts whose upstream stage used exactly one model
  SELECT draft_id FROM upstream GROUP BY draft_id HAVING COUNT(*) = 1
),
downstream AS (
  SELECT r.draft_id,
         COUNT(*)                            AS runs,
         SUM(r.status IN ('error','failed')) AS errors,
         AVG(r.duration_ms)                  AS ms
  FROM   llm_runs r
  JOIN   llm_agent_types a ON a.id = r.agent_type_id
  WHERE  a.slug = 'reviewer' AND r.draft_id IS NOT NULL
  GROUP  BY r.draft_id
)
SELECT u.upstream_model,
       COUNT(*)                          AS drafts,
       ROUND(AVG(d.runs), 2)             AS downstream_runs_per_draft,
       ROUND(100 * AVG(d.errors > 0), 2) AS pct_drafts_with_error,
       ROUND(AVG(d.ms))                  AS downstream_ms
FROM   upstream u
JOIN   uncontested c ON c.draft_id = u.draft_id
JOIN   downstream  d ON d.draft_id = u.draft_id
GROUP  BY u.upstream_model
ORDER  BY drafts DESC;
upstream_modeldraftsdownstream_runs_per_draftpct_drafts_with_errordownstream_ms
claude-haiku-4-53,8321.090.101,965
haiku (CLI)1,3181.2310.1729,630
deepseek-v4-flash1,2851.800.009,512
The whole result, from a database of 122,932 recorded calls; stage names are replaced by their role. Two different ways of going wrong: one upstream model leaves the next stage retrying 65% more often, another leaves it erroring a hundred times more and running fifteen times slower. Read it as it stands and you would go and change a model. That would be a mistake, and the next column over is what tells you so.
conclusions the data refused what survived
the questionDid the model I picked at one step make a later step worse?You group drafts by the model the Matcher ran on, and read the Reviewer’s error rate.you would have concluded0.10% → 10.17%. A hundredfold. Damning.The Reviewer’s own model changed in the same window. Nothing is being held constant.So you add the downstream model as a column, and look again.you would have concludedThe failing rows all use short model aliases rather than full provider paths.Rolled up, alias runs faillessoften — 3.09% against 6.19%. The theory is dead.So you ask what is actually driving the 6.19%.you would have concludedA web-search step failing on the network 19.62% of the time. Not a model at all.Which means every comparison that averaged it in was measuring the wrong thing.What is left once the stage, the prompt version and the week are all held fixedTwo pairs. Out of 122,932 runs.Matchersame week, same prompttwo routes to the same model family13,505 ms vs 1,594 msidentical error rate; one is 8× slowerClassifiersame week, same prompta larger model against a smaller one1,980 ms vs 5,821 ms3× faster, 2.4× the cost, both cleanlive queries against a pf-core tracking database · 2026-09-07 · agent names replaced by their role
Each of those three conclusions is one column away from looking solid, and each is wrong. The reason they can be checked at all is that every call is one row carrying its stage, its model, its prompt version, its pipeline, its status, its latency and its cost — so the follow-up question is a join rather than a re-run. The uncomfortable finding is the last one: almost nothing in a real pipeline is a controlled experiment unless you arranged for it to be, and without records at this grain you would never have learned that about your own system.

The answer this particular database gives is uncomfortable, and it is the useful kind of uncomfortable. Three times in a row the obvious conclusion was one column away from surviving, and one column away from being wrong. What finally held up was small: two comparisons where the stage, the prompt version and the week were genuinely fixed, out of 122,932 recorded calls.

That is not an argument that the tooling is clever. It is an argument that a system running unattended will hand you a confident, wrong answer unless something is keeping records at this grain — and that the same records are what tell you which of your comparisons you are allowed to believe.