Skip to content

How-to: tune ranking safely

Use this playbook to adjust ranking behavior without losing auditability.

Before you start

  • Decide the goal: one primary KPI + 2–5 guardrails
  • Ensure you can measure (stable request_id and exposure/outcome logs)

Start points:

Step 1 — Understand the current ranking contract

Read the core behavior and determinism rules:

Step 2 — Choose the smallest knob

Prefer the smallest change that can be rolled back cleanly:

  1. Config/rules change (no code)
  2. Use weights/limits/flags per tenant and merchandising constraints.

  3. Best for quick iteration and safe rollback.

  4. Pipeline change (signals/data)

  5. Add or adjust a signal end-to-end.

  6. Ranking code change

  7. Use only when the scoring/merge logic must change.
  8. Requires a stricter evaluation and review.

Step 3 — Create a candidate and keep it reproducible

  • Record the baseline version (config/rules/algo versions)
  • Record the candidate version (exact diffs)
  • Keep artifacts immutable (avoid "silent" rewrites)

Helpful reading:

Step 4 — Run offline evaluation gates

Run an evaluation report and interpret it as a decision artifact:

Step 5 — Validate determinism and joinability

These prevent "it worked on my laptop" outcomes:

Step 6 — Ship with rollback discipline

  • Run at least one rollback drill before you need it.
  • Write the decision and link it to the report and evidence kit.

Start here: