/

Engineering

How we deliver every event exactly once

Retries are not the enemy. Writing the same row twice is. Here is the ledger that sits between your source and your warehouse.

Ines Duarte

Founding engineer

7 min

Every pipeline promises at least once delivery. That sounds safe until a network blip turns one payment into two rows and a finance analyst spends a morning finding out why revenue is up 0.4 percent.

The ledger

Each event that enters Sluice gets a stable key built from its source position. Before we write to a destination we check that key against a ledger stored next to the destination itself, inside the same transaction when the destination supports it.

When the destination has no transactions

Object stores and some warehouses cannot commit a ledger entry and a data write atomically. For those we write to a staging location, record the batch, then promote it. A crash at any step leaves either nothing or everything.

What it costs

The ledger adds a few milliseconds per batch. In exchange you can delete the dedupe jobs, the window functions and the 3am pages.

Create a free website with Framer, the website builder loved by startups, designers and agencies.