How we deliver every event exactly once
Retries are not the enemy. Writing the same row twice is. Here is the ledger that sits between your source and your warehouse.
Ines Duarte
Founding engineer
7 min
Every pipeline promises at least once delivery. That sounds safe until a network blip turns one payment into two rows and a finance analyst spends a morning finding out why revenue is up 0.4 percent.
The ledger
Each event that enters Sluice gets a stable key built from its source position. Before we write to a destination we check that key against a ledger stored next to the destination itself, inside the same transaction when the destination supports it.
When the destination has no transactions
Object stores and some warehouses cannot commit a ledger entry and a data write atomically. For those we write to a staging location, record the batch, then promote it. A crash at any step leaves either nothing or everything.
What it costs
The ledger adds a few milliseconds per batch. In exchange you can delete the dedupe jobs, the window functions and the 3am pages.