Skip to content

Evaluation validity

Evaluation numbers are only useful when you can explain what they mean, what they do not mean, and what you would do if they move.

This page makes the evaluation claims in this documentation precise and conservative.

What RecSys guarantees about evaluation

RecSys can guarantee process quality (repeatability and auditability), not business outcomes.

  • Reproducible offline runs: given the same inputs and the same artifact/manifest versions, you can re-run offline evaluation and obtain the same outputs.
  • Auditable joins: exposures, outcomes, and assignments are joined by request_id with explicit join logic.
  • Decision trail: reports are intended to be stored alongside the change that produced them, so “why did we ship?” is answerable.

See also: Guarantees and non-goals.

What RecSys does not guarantee

  • That offline uplift translates to online uplift.
  • That an online experiment result generalizes beyond its traffic slice.
  • That any particular metric is “the metric.”

RecSys helps you measure and decide; it does not remove the need for product judgment.

The 4 failure modes of evaluation

1) Unjoinable logs

Symptoms:

  • Join-rate is near zero.
  • Metrics look “too good” or “too empty.”

Fix:

2) Leaky or biased offline datasets

Common causes:

  • Training/eval data includes signals that encode outcomes (data leakage).
  • Offline dataset does not represent production traffic.
  • The candidate set differs from what serving would produce.

Mitigations:

  • Treat offline evaluation as a gate (reject obvious regressions), not as the final word.
  • Keep an “offline validity checklist” in your evaluation runbook.

3) Metric mismatch

Common causes:

  • Optimizing proxy metrics that do not align with business goals.
  • Ignoring guardrails (diversity, latency, bounce, refunds, etc.).

Mitigations:

4) Overconfidence from small samples

Common causes:

  • Shipping based on tiny traffic slices.
  • Multiple comparisons without discipline.

Mitigations:

  • Prefer conservative thresholds.
  • Use interleaving or OPE only when you can justify assumptions.

References:

Use this ladder to move from “safe to merge” to “safe to ship”:

  1. Determinism + invariants (CI gate)
  2. Verify determinism: Verify determinism

  3. Pipelines invariants: Pipelines operational invariants (safety model)

  4. Offline gate (reject obvious regressions)

  5. Workflow: Workflow: Offline gate in CI

  6. Online validation (confirm impact in production)

  7. Workflow: Workflow: Online A/B analysis in production

  8. Decision + documentation

  9. Suite how-to: How-to: run evaluation and make ship decisions