Skip to content

Limitations and risks

This page explains the common failure modes when evaluating and operating RecSys.

Scope

For the blunt list of product boundaries (what is not implemented or intentionally out of scope), see Known limitations and non-goals.

Product boundaries (what you may hit quickly)

Known current boundaries include:

  • Tenant creation is DB-only today (no tenant-create admin endpoint yet)
  • Pipelines manifest registry is filesystem-based by default
  • Kafka ingestion is scaffolded but not implemented as a streaming consumer

Source of truth:

Evaluation risks (how pilots go wrong)

You can serve results but cannot measure impact

Symptoms:

  • No stable join key (request_id) across exposure and outcome logs
  • Low join rate (outcomes cannot be attributed to what was shown)
  • Reports exist but are not trusted by stakeholders

What to do:

Metrics are treated as truth instead of signals

Offline and online metrics have limits. A single metric rarely tells the whole story.

What to read:

Data risks (privacy, retention, and accidental leakage)

Typical risks:

  • Logging raw identifiers where pseudonymous IDs would suffice
  • Keeping event logs longer than necessary
  • Mixing customer tenants in the same storage without clear boundaries

What to read:

Operational risks (ship/rollback discipline)

Typical risks:

  • No rollback drill (the lever is untested until it is urgent)
  • Treating artifacts/manifests as mutable state (breaking reproducibility)
  • Running pipelines without freshness and limit guardrails

What to read:

Licensing and procurement risks

Typical risks:

  • Starting a pilot without clarifying the license path (AGPL vs commercial)
  • Procurement starts late and blocks shipping even after a successful pilot

What to do: