How-to: Run incremental pipelines

Incremental runs use a checkpoint so you can process only new days.

Prerequisites

• checkpoint_dir configured (defaults to .out/checkpoints)
• raw_source configured

Run

recsys-pipelines run \
  --config configs/env/local.json \
  --tenant demo \
  --surface home \
  --end 2026-02-01 \
  --incremental

First run: pass --start once to seed the checkpoint:

recsys-pipelines run \
  --config configs/env/local.json \
  --tenant demo \
  --surface home \
  --start 2026-01-01 \
  --end 2026-01-07 \
  --incremental

After each successful day, the checkpoint is updated automatically.

Read next

• Operate pipelines daily: How-to: Operate pipelines daily
• Schedule pipelines: How-to: Schedule pipelines with CronJob
• SLOs and freshness: SLOs and freshness
• Output layout: Output layout (local filesystem)
• Config reference: Config reference