---
title: "The worker died. The sync didn't."
description: "Kill the worker mid-batch and follow one contact: which cursor was saved, why the replayed write does not land twice, who owns the sync while it restarts. Then the two incidents that showed recovery was only half the job."
url: https://tajo.io/blog/worker-died-sync-didnt/
markdown: https://tajo.io/blog/worker-died-sync-didnt.md
lang: en-US
author: "Set Noa"
author_url: https://tajo.io/set-noa/
published: 2026-11-05
updated: 2026-11-05
alternates: {bg-BG: /bg/blog/worker-died-sync-didnt.md, cs-CZ: /cz/blog/worker-died-sync-didnt.md, de-DE: /de/blog/worker-died-sync-didnt.md, da-DK: /dk/blog/worker-died-sync-didnt.md, es-ES: /es/blog/worker-died-sync-didnt.md, fr-FR: /fr/blog/worker-died-sync-didnt.md, el-GR: /gr/blog/worker-died-sync-didnt.md, hi-IN: /hi/blog/worker-died-sync-didnt.md, id-ID: /in/blog/worker-died-sync-didnt.md, hr-HR: /hr/blog/worker-died-sync-didnt.md, hu-HU: /hu/blog/worker-died-sync-didnt.md, it-IT: /it/blog/worker-died-sync-didnt.md, ja-JP: /jp/blog/worker-died-sync-didnt.md, ko-KR: /kr/blog/worker-died-sync-didnt.md, ar-SA: /ar/blog/worker-died-sync-didnt.md, nl-NL: /nl/blog/worker-died-sync-didnt.md, no-NO: /no/blog/worker-died-sync-didnt.md, pl-PL: /pl/blog/worker-died-sync-didnt.md, pt-BR: /pt/blog/worker-died-sync-didnt.md, ro-RO: /ro/blog/worker-died-sync-didnt.md, ru-RU: /ru/blog/worker-died-sync-didnt.md, sv-SE: /se/blog/worker-died-sync-didnt.md, zh-CN: /zh-cn/blog/worker-died-sync-didnt.md, sl-SI: /sl/blog/worker-died-sync-didnt.md, sr-Latn-RS: /sr/blog/worker-died-sync-didnt.md, th-TH: /th/blog/worker-died-sync-didnt.md, tr-TR: /tr/blog/worker-died-sync-didnt.md, ur-PK: /ur/blog/worker-died-sync-didnt.md, vi-VN: /vi/blog/worker-died-sync-didnt.md, sk-SK: /sk/blog/worker-died-sync-didnt.md}
---

[Data Sync](https://tajo.io/blog/?category=data-sync)

# The worker died. The sync didn't.

Kill the worker mid-batch and follow one contact: which cursor was saved, why the replayed write does not land twice, who owns the sync while it restarts. Then the two incidents that showed recovery was only half the job.

sync failure recovery

What happens to a running sync when the worker dies in the middle of a batch?

The cursor moves only at the end of a clean batch, so a worker killed mid-batch leaves it at the start of that batch and the next run reads the same page. Each record's delivery key is derived from the page cursor and its position, and a seven-day ledger of completed writes stops the replay from landing twice. One lease per sync keeps two runs from working at once. That was the easy half. One sync kept failing, run after run, before anything paused it, and a workspace sat silent with errors and no success before anything noticed. So there is now a breaker that pauses after five consecutive failures and an hourly alert on twenty-four hours without a success.

[Learn more](https://tajo.io/blog/worker-died-sync-didnt/#follow-one-contact)

In a [talk published on 3 October 2026 by AI Engineer](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=701s), a Crusoe developer advocate launched a PyTorch training job on two A100 nodes. Then the advocate told the demo’s own monitoring system that one GPU had thrown an XID 79 error. The node was marked down and the job cancelled. The process got a SIGTERM and “[up to two minutes to handle that sig term to either save your checkpoints or flush your logs](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=633s)”. A spare node came in, and “[the slurm job will restart. It’ll load the checkpoint and continue training where it left off](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=769s)”. A little under fifteen minutes end to end. Nobody was paged.

A sync is a much smaller animal than a training run. The question is the same. What does a run need so that killing the worker costs minutes and no duplicates?

This is the answer for [Tajo’s own sync runner](https://tajo.io/resources/help/troubleshooting/recover-a-stuck-or-failed-sync/), read from the code rather than from a brochure. [Reverse ETL](https://tajo.io/blog/what-is-reverse-etl/) describes the shape of the job. This is what happens when the job dies.

## Follow one contact

A run reads the source one page at a time, from a saved cursor. Call the page cursor `c3` and our contact the 212th record on that page. Three things are true about that contact before anything is written.

First, it has a delivery key. The key is a hash of the sync’s id, the source, the page cursor, the position on the page and the record itself. The cursor and the record go into the hash and are never embedded in the key, so the key carries [no email address](https://tajo.io/resources/help/getting-started/understanding-data-sync/).

Second, the write goes through a [ledger](https://tajo.io/resources/help/troubleshooting/recover-a-stuck-or-failed-sync/#ledger), not straight to the destination. Before the destination is called, the key is claimed as pending. When the write succeeds, the row is marked completed and kept for seven days.

Third, the cursor has not moved. It moves only at the end of a batch, only if every record in that batch landed, and only after the run has proved it still owns the sync.

Now kill the worker between the write and the checkpoint.

On a SIGTERM every queue worker drains under one shared fifteen-second deadline, and the process exits after twenty-five seconds whatever is still running; systemd gives it forty-five before it kills. A run that did not finish inside that window is still marked in flight, and the next worker picks it up through the queue’s stalled-job recovery. On a hard kill there is no drain at all. Either way the same two facts hold. The destination has our contact. Our [cursor](https://tajo.io/resources/help/troubleshooting/recover-a-stuck-or-failed-sync/#cursor) still says `c3`.

## The replay

The next run starts at `c3`, reads the same page, derives the same key for the 212th record, and asks the ledger. The ledger says completed. The destination is not called; the stored result is replayed and the run moves on. On the bulk path, where Brevo’s import endpoint takes a whole page in one call, the same check runs per key before the request is built. A page in which every key is already completed sends nothing.

That is the whole trick, and it is worth saying why it works. The key is derived from the page cursor at the start of the batch, not from the run that happened to be executing. A crash, a retry sixty seconds later and a scheduled run tomorrow all [derive the same key](https://tajo.io/resources/help/troubleshooting/recover-a-stuck-or-failed-sync/#ledger) for the same record. The ledger does not care which of them is asking.

Now the cost. The [ledger forgets after seven days](https://tajo.io/resources/help/troubleshooting/recover-a-stuck-or-failed-sync/#ledger). The code’s own comment says seven days covers a delayed retry and a short outage, and that is all it claims. A replay from a cursor older than that writes again. And a run that is interrupted hard can wait. The dead run’s lease holds for up to fifteen minutes, and on a scheduled sync a run whose lease is free is [swept and rescheduled only once it started over an hour ago](https://tajo.io/resources/help/troubleshooting/recover-a-stuck-or-failed-sync/#stuck-runs). Crusoe’s demo is [back under fifteen minutes](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=753s). Ours is not always.

## Who owns the sync while it restarts

One run owns a sync at a time. The run takes a [lease](https://tajo.io/resources/help/troubleshooting/recover-a-stuck-or-failed-sync/#lease) in Redis with its own token, a fifteen-minute life and a renewal every five minutes. Before every checkpoint it asserts that it still holds that lease. If it does not, it stops. The run is recorded with the [plain sentence the person will read](https://tajo.io/resources/help/troubleshooting/recover-a-stuck-or-failed-sync/#failures): “The run lost its slot part-way through and stopped to avoid writing twice.” A second run that collides with a live owner is not retried. It [ends with](https://tajo.io/resources/help/troubleshooting/recover-a-stuck-or-failed-sync/#failures) “Another run of this sync was already in progress”, and the next scheduled run starts normally.

Ownership of the fix is written down too. Every [failure code](https://tajo.io/resources/help/troubleshooting/recover-a-stuck-or-failed-sync/#failures) maps to one sentence on what happened, one on what to do next, and an owner: you or Tajo. A destination that rejected every record because the mapping is wrong is yours. A [lost lease is ours](https://tajo.io/resources/help/troubleshooting/recover-a-stuck-or-failed-sync/#failures). The run’s page shows the next step; the [sync recovery guide](https://tajo.io/resources/help/troubleshooting/recover-a-stuck-or-failed-sync/#failures) lists which.

So far this is the easy half. A crash costs one replay and nothing lands twice. Two incidents taught the other half.

## Failure after failure

One sync failed over and over. Not in a tight loop: once per scheduler interval, run after run, with the same message every time, that a custom id cannot contain a colon. The run never reached the destination. It died while being queued, before any runner saw it, so an alert keyed on the runner never had anything to key on.

Nothing watched the streak. That is the whole finding.

The fix is a breaker. After every recorded failure it reads the sync’s last five finished runs since the sync was last changed. If all five are errors it pauses the sync, writes one audit row with the reason `circuit_breaker_tripped`, and fires one critical alert. The streak is derived from the stored runs, not from a counter in the worker. A failure recorded by any path counts, including the queueing failure that caused the incident. The code’s comment says it would have stopped that incident at run five. One success resets the streak. Un-pausing re-arms the breaker instead of re-tripping it on the old rows. The [Shopify connector guide](https://tajo.io/docs/ecommerce-platforms/shopify/) says the same thing in one line: after five consecutive failed runs the sync pauses itself and its page shows the last error.

What the breaker does not do is fix anything. Crusoe replaces the node and restarts the job without asking. Tajo stops and asks: a paused sync stays paused until a person fixes the cause and [publishes it again](https://tajo.io/resources/help/troubleshooting/recover-a-stuck-or-failed-sync/#pause). The cautious default is deliberate: an automatic retry of a sync that writes to customer records is itself a write.

## The silence nobody saw

The objection is fair, and it goes like this: “A breaker that pauses after five failures would have caught the second incident too. Why build a second watcher?” Because the second incident did not look like failure. For a long stretch a workspace had an active sync, a growing pile of error rows and not one success, ever, and nothing alerted. Every signal that existed was keyed on activity: a run failed, a webhook arrived, a budget was crossed. Zero successes is not an event. It is the absence of one, and a breaker only runs when a failure is recorded. A scheduler that is dead, a queue that is wedged, a worker that never boots: these record nothing, and a breaker never wakes.

A locomotive’s dead-man’s handle works the other way round. The driver has to keep pressing it, and silence stops the train. The sync version [runs once an hour](https://tajo.io/resources/help/troubleshooting/recover-a-stuck-or-failed-sync/#hourly-check), at thirteen minutes past. For every workspace with an active sync it asks one question: has any run succeeded in the last twenty-four hours? A workspace whose oldest active sync was enabled less than a day ago is left alone; it has not had a day to succeed yet. One that is older and silent gets one critical alert. The alert says which silence it is: runs recorded but none succeeding, or no runs recorded at all. It fires once per incident. A marker in Redis holds it closed until the first successful run, then clears, so the next incident alerts again.

Both watchers landed in one commit, after the fact. The breaker’s comment says it would have stopped the first incident at run five. The dead-man’s comment says a sync enabled long ago with zero runs must alert. Neither existed when it was needed. We wrote them afterwards, with both incidents recorded in the file header so nobody forgets why.

## What is built, and what is not

Built, as of this writing:

- The cursor is saved at the end of every clean batch, and again on a crash.
- Every record’s delivery key is derived from the page cursor and its position, and checked against a [seven-day ledger](https://tajo.io/resources/help/troubleshooting/recover-a-stuck-or-failed-sync/#ledger) before each destination call.
- One [lease](https://tajo.io/resources/help/troubleshooting/recover-a-stuck-or-failed-sync/#lease) per sync, with a fifteen-minute life.
- Five [attempts](https://tajo.io/resources/help/troubleshooting/recover-a-stuck-or-failed-sync/#retries) per run, backing off from sixty seconds.
- A [breaker](https://tajo.io/resources/help/getting-started/understanding-data-sync/) that pauses the sync after five consecutive failures and records why.
- An [hourly alert](https://tajo.io/resources/help/troubleshooting/recover-a-stuck-or-failed-sync/#hourly-check) on twenty-four hours without a success.

Not built, and worth knowing before you rely on it:

- The dead-man watches the workspace, not the sync. One healthy sync hides a silent sibling in the same workspace.
- Its alert [reaches Tajo’s operators](https://tajo.io/resources/help/troubleshooting/recover-a-stuck-or-failed-sync/#hourly-check): a log line always, and Sentry, a webhook or an email where configured. It does not appear on the sync’s page. The breaker’s pause does.
- Nothing restarts a paused sync. A person does.
- Recovery after a hard kill is not fast. The lease can hold for fifteen minutes and [the sweep waits an hour](https://tajo.io/resources/help/troubleshooting/recover-a-stuck-or-failed-sync/#stuck-runs).
- Both thresholds, five failures and twenty-four hours, are [deployment settings](https://tajo.io/resources/help/troubleshooting/recover-a-stuck-or-failed-sync/#limits). You cannot set them per sync.

[A long chain of nearly perfect steps still fails](https://tajo.io/blog/almost-perfect-steps-unreliable-chain/); the way out is to define the end state and check for it. This article is the operational half of that. Define what a dead run leaves behind, and check for the nothing that silence leaves behind.

Crusoe’s closing line was that “[failures are inevitable](https://www.youtube.com/watch?v=bRGyYaE0lxI&t=955s)”. Ours is one step on. The crash was the easy half, because it left rows. The silence was the half we had to go looking for.

## Frequently Asked Questions

**What happens to a running sync when the worker dies in the middle of a batch?**

The cursor still points at the start of the interrupted batch, because Tajo moves it only after a batch lands cleanly. The next run reads the same page again. Each record carries a delivery key derived from the page cursor and its position, and a ledger keeps every completed write for seven days, so records already written are replayed from the ledger instead of being sent to the destination a second time.

**Why does a replayed sync not write the same contact twice?**

Before every destination call the run checks the record's delivery key against a ledger. The key is a hash of the sync, the source, the page cursor, the position on the page and the record itself, so a crash, a retry a minute later and a scheduled run the next day derive the same key for the same record. A key already marked completed inside the seven-day window is not sent again.

**What is a dead-man alert for a sync?**

A check that fires on absence rather than on failure. Once an hour Tajo looks at every workspace with an active sync and alerts when no run has succeeded in the last twenty-four hours, including the case where no run was recorded at all. It fires once per incident and re-arms after the first successful run. It exists because a failure alert can only see failures that were recorded.

## Related Articles

### [Six rules for a sync that runs while nobody watches](https://tajo.io/blog/six-rules-unattended-sync/)

Nov 6, 2026 · 8 min read

7 sections

### [The workflow between the boxes](https://tajo.io/blog/the-workflow-between-the-boxes/)

Oct 8, 2026 · 4 min read

Store record → Operator correction → Email destination

### [Brevo integrations: choose a connection you can verify](https://tajo.io/blog/brevo-connector-guide/)

Aug 19, 2026 · 5 min read

6 sections