stuck heads starve the tail in newest-first queues
the job reported zero while the backlog grew, teaching me that ambiguous states are not transient and sorted queues freeze the line.
what it looked like from the outside
i pushed the backfill script and went to sleep. it ran in the maintenance window, finished with exit code 0, and logged 'sync complete'. the metrics showed the queue depth at zero. but the morning reconciliation found a gap of about a thousand records in the ledger. they weren't scattered; they were a solid block of missing time. from the outside, it looked like a source system timeout or a data retention policy kicking in at the vendor level. the logs were clean. no stack traces, no rate limit alerts, nothing but the heartbeat. it felt like a ghost disappearance, the kind of thing you file a ticket for and ignore because it usually doesn't recur. the system thought it was healthy, but the business state was incorrect.
the wrong theory
i assumed the issue was a transient network glitch that resolved itself before the timeout fired, causing the cursor to skip a chunk of data. at scale, i see this constantly where a tcp drop causes a silent ack but an incomplete response. i thought the 'three-way split' in my processor--sorting results into success, retry, and failure buckets--was handling the edge cases. the 'ambiguous' bucket was throwing items back into the queue with a jittered delay. i thought the logic was sound because the retry counter was incrementing in the logs. i believed the new batch would eventually catch the stragglers once the congestion cleared. the symptoms of a temporarily overloaded pipe and a permanently blocked logic path looked exactly the same in the aggregate metrics: high retry count, then silence.
what actually settled it
i stopped trusting the aggregate counters and instrumented the worker to dump the raw json payload from the source to a local file before it ran any schema validation or queue insertion. i ran the next cycle and waited for it to 'complete'. when i checked the dump, the file was heavy--plenty of records. but the first record in the list had a slightly malformed timestamp that the parser couldn't coerce. because the retry queue was sorted 'newest-first', and the malformation happened during ingestion, the poisoned record kept getting bubbled to the top of the stack. the worker would pick it up, fail to parse, mark it for retry, and burn its compute budget on that single row. the thousand valid records behind it in the list never got touched. the job wasn't finishing because the work was done; it was finishing because the worker ran out of time trying to fix the same broken record.
the fix
the fix required breaking the assumption that 'ambiguous' means 'fixable on the next pass'. i added a hard 'attempt cap' to the query that feeds the worker, so any row with a retry count over three is physically excluded from the select statement. those go to a dead letter queue for human inspection instead of clogging the pipe. i switched the retry lane from a sorted list to a round-robin shuffle. now, if one record is bad, it doesn't sit at the head of the line blocking the others. i also added a sanity check: if the transport returns a a few hundred but the processed item count is zero, the job crashes instead of exiting clean. it forces a look at the data shape rather than assuming success.
the rule i keep now
i treat every sorted queue as a potential single point of failure now. if you order the work and the head gets stuck, the tail starves--it's a fundamental property of queues, not a bug in the code. whenever i write a retry loop, i have to decide if the error is transient--a network blip--or permanent--a data shape mismatch. mixing them in the same bucket is fatal. i shuffle the retries to guarantee fairness, and i fail fast on zero-processed batches. the silence of a successful exit code is the most dangerous signal in the stack, so i make sure it’s never lying to me.
the rule: a clean exit code on a stuck head is the most dangerous kind of success.