A composite example, assembled from failure patterns that show up again and again in production LLM systems:
A support agent ships with a 94 percent pass rate on a 200-case golden set, two points better than the previous version. Three weeks later it quotes a return window to a customer with complete confidence. The policy changed in September. Every run that week finished completed. Rerun the suite and it still passes, because the golden answer for that case is the old policy too.
completed wrong answer, delivered confidently
Nothing in that pipeline was broken. The suite was measuring a distribution that no longer existed, and nothing was watching the gap. We'll come back to exactly what would have caught it.
Four kinds of drift, one green dashboard
An offline eval set is a sample from a joint distribution P(x, y), taken once and scored by one grader. Production moves every part of that.
Four things move under a fixed eval set. Concept drift did the damage in the opening example.
Covariate shift (data drift). P(x) changes. Users ask about shipping in November and billing in March, paste screenshots, ask two questions at once. The golden set was written by the people who built the system, so it was never a sample of real traffic in the first place. An aggregate pass rate can't reveal a mix change because it averages over the old mix.
Concept drift. P(y | x) changes: same question, new correct answer. Prices, policies and docs move, and every golden label that depends on them now rewards the wrong output. The return window is concept drift, and it's the worst kind because the regression suite actively penalizes the fix.
Upstream drift. The system changes without a deploy. Chen, Zaharia and Zou compared the March and June 2023 GPT-4 snapshots and found the June model followed formatting and chain-of-thought instructions noticeably less reliably. Their headline prime-number result is contested: Narayanan and Kapoor pointed out the original test set contained only primes, so much of the "drop" was a shift in the model's default answer rather than lost capability. That's still a production problem. A behavior change on an alias you don't pin breaks downstream parsers just as hard as a capability loss. The same applies to the components around the model: a re-chunked index, a changed tool schema, a library that moved a major version. That last one is training-serving skew, and we've written about it before. Sculley et al. listed "changes in the external world" as a core source of ML technical debt a decade ago, and LLM systems have more external surface than most.
Criteria drift. Your definition of correct changes. Shankar et al. watched engineers build LLM graders and found they couldn't write a rubric until they had graded outputs, and grading outputs kept changing the rubric.
On top of all four sits Goodhart's law. Every prompt change you validate against the golden set fits the prompt a little more tightly to those 200 cases. Eventually the suite is partly a training set.
“None of this trips a status check. The run completes, latency is fine, and the answer is wrong.
Catch concept drift at the source
Concept drift is the one that hurt in the example, and none of the usual tools catch it. The router still sends the question to the same intent, so input-distribution checks stay quiet. The gate still passes because the golden label is stale too. Even a faithfulness check passes if the knowledge base wasn't updated: the answer is perfectly faithful to an outdated document.
The fix is provenance. Every golden case records the IDs and content hashes of the source documents its expected answer depends on. A nightly job compares those hashes against the current knowledge base. When a policy doc changes, every case that cites it is marked stale and pulled from the pass rate until someone relabels it. The same job checks that the KB itself is fresh against its system of record, because a stale index makes faithfulness scoring meaningless.
That turns "the return policy changed" from something you learn from a customer into a relabeling ticket filed the night the doc changed.
Measure covariate shift without labels
Input drift needs no labels. Most support and ops agents already route by intent, either with a Document Classification node or a router step. Compare the router's label distribution over a sample of last week's production runs against the golden set's, using the population stability index from credit-risk modeling:
import random
from collections import Counter
from datetime import datetime, timedelta, timezone
import numpy as np
from strongly import Strongly
client = Strongly()
WF, ROUTER_NODE = "wf_support_agent", "classify-intent"
rng = random.Random(7)
def production_intents(days: int = 7, max_runs: int = 2000) -> Counter:
since = (datetime.now(timezone.utc) - timedelta(days=days)).isoformat()
runs = list(client.executions.list(workflow_id=WF, since=since, status="completed"))
counts = Counter()
for run in rng.sample(runs, min(max_runs, len(runs))): # sample first: one span call per run
for span in client.executions.list_spans(run.id, node_id=ROUTER_NODE):
counts[span.output_data["label"]] += 1
return counts
def psi(expected: Counter, actual: Counter, eps: float = 1e-4) -> dict:
labels = sorted(expected.keys() | actual.keys())
e = np.array([expected[k] for k in labels], float); e = e / e.sum() + eps
a = np.array([actual[k] for k in labels], float); a = a / a.sum() + eps
contrib = (a - e) * np.log(a / e)
return {"psi": float(contrib.sum()), "by_label": dict(zip(labels, contrib.round(3)))}The common thresholds (under 0.1 stable, above 0.25 a real shift) come from credit scoring with ten roughly equal bins and large samples. Your bins are intents, some of them rare, and your golden set is small, so a single sparse label can move the score a lot. Treat the per-label contributions as the useful output: they tell you which slice to write cases for next. Also remember the router is a model too. If its own behavior drifts, PSI moves for the wrong reason, so pin its version and spot-check its labels.
If the router is a classical classifier served from the Strongly Model Registry, you don't need the script above. Strongly's drift detection is built for traditional ML models like that one; it doesn't watch the LLM, which is why the rest of this post exists. The platform records the predictions it serves, and drift detection compares a production window against the model's reference baseline on a schedule, notifying the owner in-app and optionally by email or Slack when a result reaches warning or alert:
client.drift_detection.update_settings(
ROUTER_MODEL_ID,
enabled=True,
schedule={"enabled": True, "frequency": "daily", "timeWindowDays": 7},
notifications={"email": True, "slack": SLACK_URL},
performance_drop_warning=0.05, # relative to baseline accuracy
performance_drop_alert=0.10,
)
run = client.drift_detection.analyze(model_id=ROUTER_MODEL_ID, window_days=7) # on demandCustom tests, including PSI over the intent label, can be registered with client.drift_detection.create_algorithm and enabled per model with their own warning and alert thresholds.
Feed the eval set from traces
The supply line for new cases is production traces. On Strongly, two facts make that practical. Every execution records whether it ran in production or development and which workflow version it ran, so you can separate "we shipped a regression" from "the world moved." And every model and tool call is a span with its full payload: agent runs carry an llm:<model> span per model call (messages, reply, tool calls, token usage) and a tool:<name> span per tool call, marked failed when the tool returned an error.
A workflow's tracing level (settings.spanLevel) at minimal drops successful LLM and tool spans, and at error it keeps only failures. That's sensible for high-volume ETL. On anything you evaluate, it deletes exactly the evidence you need: the successful-looking wrong answer.
Sample three ways. A uniform sample estimates current behavior. A stratified sample guarantees rare slices show up; a 3 percent segment otherwise appears once in a sample of fifty. A targeted sample finds failures fast. Two kinds of signal feed it. Platform signals: runs ending partial_success or completed_with_gaps, slow runs, failed tool spans, Human Checkpoint rejections. Implicit feedback: escalations to a human, a user re-asking the same question within minutes, thumbs-down. Implicit feedback is the cheapest labeled data you'll ever get, because users already did the labeling. For agents you can find some of it without leaving Strongly: executions.list(search=...) matches text in agent transcripts, so phrases like "speak to a human" or "that's not what I asked" become a query.
since = (datetime.now(timezone.utc) - timedelta(days=1)).isoformat()
runs = [r for r in client.executions.list(workflow_id=WF, since=since) if r.is_production]
by_duration = sorted((r for r in runs if r.duration_ms), key=lambda r: r.duration_ms)
picked = {r.id: r for r in (
rng.sample(runs, min(40, len(runs)))
+ by_duration[int(len(by_duration) * 0.9):]
+ [r for r in runs if r.status in ("partial_success", "completed_with_gaps")]
+ [r for r in runs if r.id in escalated_execution_ids] # from your helpdesk
+ list(client.executions.list(workflow_id=WF, since=since, search="speak to a human", limit=50))
)}
def to_review_item(run) -> dict:
spans = client.executions.list_spans(run.id)
return {
"execution_id": run.id,
"version": run.workflow_version_number,
"llm_calls": [(s.input_data, s.output_data) for s in spans if s.name.startswith("llm:")],
"failed_tools": [s.name for s in spans if s.name.startswith("tool:") and s.status == "failed"],
"tokens": client.executions.ai_usage(run.id).total_tokens,
}
review_queue = [to_review_item(r) for r in picked.values()]Redact before the queue leaves the platform; the Guardrails and PII Redactor nodes do that inside a workflow. Then a person reads it. Hamel Husain's stopping rule is the right one: keep reading until you stop learning anything new. Give each failure a short label. After a few dozen traces those labels converge into a handful of failure modes, and those become your eval dimensions.
Promote cases into three datasets, versioned, with the dataset version stored next to every score:
Append-only. Confirmed incidents go in. Stale cases (from provenance) are retired explicitly and replaced.
Rebuilt monthly from current traffic. Its scores aren't comparable across rebuilds.
Never used for prompt iteration. It exists only to measure how much the other sets have been overfit. That's the direct answer to Goodhart.
Your judge is a classifier. Validate it like one.
Zheng et al. found strong LLM judges agreed with human preferences over 80 percent of the time, about the rate humans agree with each other, and in the same paper documented position, verbosity and self-enhancement bias. Wang et al. showed that reordering two answers was enough to let Vicuna-13B beat ChatGPT on 66 of 80 queries with ChatGPT judging. In expert domains, Szymanski et al. measured 60 to 68 percent judge-expert agreement in dietetics and mental health, compared with 72 to 75 percent between experts.
Agreement in expert domains (dietetics and mental health), Szymanski et al. Bars show the top of each range.
So the judge gets its own labeled data and its own metrics. One domain expert labels 100 to 200 sampled traces pass or fail, each with a one-line critique. Binary labels are more consistent than 1-to-5 scales and give you a confusion matrix; Hamel Husain and Eugene Yan both arrived at this independently. Split the labels into dev and holdout, tune the judge prompt on dev using the critiques as few-shot examples, and report on holdout. For pairwise comparisons, run every pair in both orders and only count verdicts that agree.
On Strongly the judge is an evaluation node (LLM as Judge, Pairwise Comparator, Faithfulness Checker) inside a small draft workflow, pinned to one AI Gateway model. Drafts run one execution at a time by default; raise max_concurrent_executions (up to 10) so judging 200 traces doesn't take half an hour.
from concurrent.futures import ThreadPoolExecutor
from sklearn.metrics import cohen_kappa_score, confusion_matrix
JUDGE_WF, JUDGE_NODE = "wf_support_judge", "llm-as-judge"
client.workflows.update(JUDGE_WF, max_concurrent_executions=10)
def run_to_completion(workflow_id: str, inputs: dict) -> dict:
"""Run a draft and return every node's output, keyed by node ID. Fails loudly."""
run = client.workflows.execute(workflow_id, trigger_inputs=inputs)
if run.outputs is not None: # short runs return finished
return run.outputs
progress = client.executions.progress(run.execution_id) # long-polls up to ~9s
while not progress.is_terminal:
progress = client.executions.progress(run.execution_id)
if progress.outputs is None: # failed, stopped, error, timeout
raise RuntimeError(f"{run.execution_id} ended {progress.status}: {progress.next_action}")
return progress.outputs
def judge(trace: dict) -> bool:
return run_to_completion(JUDGE_WF, {"trace": trace})[JUDGE_NODE]["passed"]
with ThreadPoolExecutor(max_workers=10) as pool:
judged = list(pool.map(judge, [item["trace"] for item in holdout]))
expert = [item["expert_pass"] for item in holdout]
tn, fp, fn, tp = confusion_matrix(expert, judged, labels=[False, True]).ravel()
print(f"TPR {tp/(tp+fn):.2f} TNR {tn/(tn+fp):.2f} kappa {cohen_kappa_score(expert, judged):.2f}")At a 90 percent base pass rate, a judge that approves everything scores 0.90 accuracy and 0.00 TNR. TNR, the rate at which the judge catches bad outputs, is what matters for monitoring. Kappa corrects for chance agreement and is what you compare across judge versions.
Then pin it: model version, prompt and holdout scores, stored together as a versioned artifact. Upgrading the judge means changing your measuring instrument, so rerun the holdout first. Re-label a fresh batch every month or two. If kappa drops, the criteria may have drifted rather than the judge.
Strongly's evaluation nodes make one design choice worth copying into any judge you write yourself. If the judge call fails or returns something that isn't a valid verdict, the node fails and quotes the raw output. It never writes a default score. A judge that maps its own errors to 0 makes a provider outage look like a quality regression, and one that maps them to "pass" hides real regressions.
Gates that know what they can't see
At p = 0.9 and n = 200, the standard error of a pass rate is about 2.1 points, so the "two points better" release in the opening was noise. Worse, the gate's blind spot is wider than most teams assume. An unpaired comparison at 200 cases, 80 percent power and α = 0.05 can only reliably detect a drop of about 10 points. Pairing the same cases across baseline and candidate helps, but not by an order of magnitude. A gate that passes because nothing reached significance is not evidence that nothing regressed. Write the minimum detectable effect next to the gate so nobody reads it that way.
Four changes make a gate honest:
Count only discordant cases (baseline passed and candidate failed, or the reverse). With the few dozen discordant pairs a 200-case suite produces, the chi-square approximation behind McNemar's test is unreliable; use the exact binomial test on the discordant counts, one-sided, with α stated.
Block when the upper confidence bound on the regression rate exceeds a margin you chose deliberately (say 2 points), not only when the regression is "significant."
Testing eight slices at α = 0.05 gives roughly a one-in-three chance that at least one fails by chance. Use Holm-Bonferroni.
Safety refusals, PII handling, past incidents. Any failure blocks the release.
For stochastic systems, run each case k times and keep the per-case pass rate rather than a majority vote. A case that passes 3 of 5 runs is a finding, not a pass. Report flaky cases separately.
import math
from scipy.stats import binomtest
def paired_gate(base: list[bool], cand: list[bool], alpha=0.05, margin=0.02) -> list[str]:
n = len(base)
broke = sum(b and not c for b, c in zip(base, cand))
fixed = sum(c and not b for b, c in zip(base, cand))
problems = []
if broke + fixed and binomtest(broke, broke + fixed, 0.5, alternative="greater").pvalue < alpha:
problems.append(f"significant regression: {broke} broke vs {fixed} fixed")
diff = (broke - fixed) / n # paired drop in pass rate
se = math.sqrt(max((broke + fixed) / n**2 - (broke - fixed) ** 2 / n**3, 0))
if diff + 1.645 * se > margin:
problems.append(f"cannot rule out a drop larger than {margin:.0%} (upper bound {diff + 1.645*se:.1%})")
return problemsBefore deploying, replay yesterday's sampled production inputs through the candidate as well. It's the cheapest form of shadow testing, because the inputs are real, recent and already redacted. A run's trigger span records the payload the run started with, so replaying one is two calls: read the trigger node's span, then hand its output_data to run_to_completion(DRAFT_WF, ...).
Everything above runs against a draft copy made with duplicate. Deployed Strongly workflows only run from their triggers, and draft runs are tagged as development, so they stay out of production metrics. A deployed workflow also can't be edited in place (the API refuses with a 409 until it's undeployed), so promotion is an explicit step: copy the tested graph onto the production workflow, commit it, deploy it, and keep the previous version ID for rollback.
previous = next(v for v in client.workflows.versions(PROD_WF) if v.is_deployed)
candidate = client.workflows.retrieve(DRAFT_WF)
client.workflows.undeploy(PROD_WF)
client.workflows.update(PROD_WF, nodes=candidate.nodes, connections=candidate.connections)
client.workflows.create_version(PROD_WF, message="refund policy grounding; gate: 0 broke, 7 fixed")
client.workflows.deploy(PROD_WF)
# rollback, if the daily monitor says so:
# client.workflows.deploy_version(PROD_WF, previous.id)Schedule the promotion for a quiet window, since the workflow is briefly undeployed. If the change is to a registry model (a router, a scorer, a classifier inside the workflow) rather than the prompt, client.ab_tests covers the online half: canary rollouts that advance only while the canary stays as healthy as the control, and experiments with a declared minimum detectable effect and sequential early stopping.
Monitoring for failures that return 200
Gates catch regressions you introduce. Drift under a version that's been live for a month needs online evaluation, which comes in three layers.
Status and latency
The Workflow Monitor shows active workflows and agents, 24-hour run volume, success rate and anything whose latest run failed. Each workflow's Statistics page adds p95 duration and recent problem runs. Workflow Alerts fire on failures, timeouts, duration thresholds, specific node-type failures, consecutive failures and dead-letter backlog. This layer is necessary and almost never where quality problems show up first.
Proxy metrics on every trace
These need no judge and cost almost nothing. Any one alone is ambiguous; correlated moves across several mean something changed.
| Signal | Source | What a shift usually means |
|---|---|---|
| Tokens per run | client.executions.ai_usage(id) | Prompt bloat, retrieval returning more, model verbosity change |
| Turns per agent run | Agent run span | Agent struggling to finish, looping on tools |
| Tool failure rate | tool: spans | Upstream API or schema change |
partial_success / completed_with_gaps rate | client.executions.stats(WF, since=...) counts by status | Nodes silently producing nothing |
| Re-ask and escalation rate | Your helpdesk | Users not getting answers |
| Human Checkpoint reject rate | Checkpoint responses | Human labels you're already collecting |
| Intent PSI | Router spans, or registry drift detection | Covariate shift |
| Per-model latency and error rate | client.ai.analytics.time_series(...) | Provider-side change, rate limiting, a silent model swap behind an alias |
Sampled judging with a control chart
Run the trace sampler daily, score the sample with the pinned judge, and treat the daily pass rate as a process to control. A p-chart from statistical process control fits: set the center line p₀ from the trailing four weeks and the control limits at p₀ ± 3√(p₀(1 − p₀)/n). Page on any day outside the lower limit, or on two consecutive days below the 2σ line (one of the Western Electric rules).
A daily p-chart catches the slow slide
The sample size sets both cost and sensitivity. At p₀ = 0.9, judging 200 runs a day puts the 3σ limit 6.4 points below center; 800 runs tightens it to 3.2 points, at four times the judge spend. Pick the smallest drop you need to catch within a day and size the sample from that.
Shorter is more sensitive: quadrupling the sample halves the drop you can detect in a day.
Strongly alerts fire on run outcomes, not metric thresholds, so the monitor reports a breach by emitting a platform event. A deployed incident workflow with an Event trigger listens for it and does the response: post to Slack or PagerDuty, open a ticket, attach the failing traces.
def daily_quality_check(judged_today: list[bool], p0: float):
n, p_hat = len(judged_today), sum(judged_today) / len(judged_today)
sigma = math.sqrt(p0 * (1 - p0) / n)
if p_hat < p0 - 3 * sigma or (p_hat < p0 - 2 * sigma and yesterday_below_2sigma()):
client.workflows.emit_event(
event_type="quality.breach",
event_data={"workflow_id": WF, "pass_rate": p_hat, "baseline": p0, "n": n},
source="daily-eval-monitor",
)Slice the same chart by workflow version. A drop confined to the newest version points at the deploy. A drop across versions that haven't changed points at concept drift, upstream drift or a dependency. Where an extra model call per run is affordable, also score inline: Strongly's reference RAG pattern sends every answer through a Faithfulness Checker and a Conditional, routing failures to a Reflection node, so every run carries a logged score.
Back to the return window
Here is what would have caught the opening incident, in order. Provenance would have flagged every return-policy case as stale the night the policy doc changed. A KB freshness check would have shown the index was behind its source. Failing that, escalation and re-ask rates on the returns intent would have moved within days, and the daily judge sample, scored against a current rubric, would have gone below its control limit soon after. The status dashboard would never have moved at all.
Each of those layers is cheap on its own. The expensive part is the habit: every confirmed production failure becomes a regression case, the rolling set gets rebuilt every month, and the judge gets re-validated on a schedule. Keep doing that and "the evals passed" starts to mean something about production.
“Every confirmed production failure becomes a regression case. Keep doing that and "the evals passed" starts to mean something about production.
Evals are a Day Two problem.
We build AI systems with tracing, calibrated evaluation and production monitoring from the first deploy, so silent failures show up on a dashboard before they show up in a customer thread. Get a free expert review of your AI idea or roadmap, read by our Chief AI Officer.
References
- Chen, L., Zaharia, M., Zou, J. (2023). How is ChatGPT's behavior changing over time? arXiv:2307.09009
- Narayanan, A., Kapoor, S. (2023). Is GPT-4 getting worse over time? AI Snake Oil
- Shankar, S., et al. (2024). Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences arXiv:2404.12272
- Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena arXiv:2306.05685
- Wang, P., et al. (2023). Large Language Models are not Fair Evaluators arXiv:2305.17926
- Szymanski, A., et al. (2024). Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks arXiv:2410.20266
- Sculley, D., et al. (2015). Hidden Technical Debt in Machine Learning Systems NeurIPS 2015
- Husain, H. Your AI Product Needs Evals
- Husain, H. Creating a LLM-as-a-Judge That Drives Business Results
- Yan, E. Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)
- Workflows: execution tracing and the Workflow Monitor
- Testing Workflows: span types and workflow statistics
- Workflow Nodes: Evaluation, Control Flow and Triggers
- Workflow Alerts
- Executions API: runs, spans, AI usage, tracing levels
- Python SDK: Workflows
- Python SDK: Executions
- Python SDK: Getting Started (response models and pagination)
- Python SDK: Drift Detection
- Python SDK: A/B Tests
- Python SDK: AI Analytics