<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Mohamed Aboelmagd]]></title><description><![CDATA[Mohamed Aboelmagd]]></description><link>https://mohamedaboelmagd.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Mon, 07 Sep 2026 04:14:13 GMT</lastBuildDate><atom:link href="https://mohamedaboelmagd.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[40001 is not a query error]]></title><description><![CDATA[The PostgreSQL manual is unusually direct about this:

When an application receives this error message, it should abort the current transaction and retry the whole transaction from the beginning.

"Th]]></description><link>https://mohamedaboelmagd.hashnode.dev/40001-is-not-a-query-error</link><guid isPermaLink="true">https://mohamedaboelmagd.hashnode.dev/40001-is-not-a-query-error</guid><category><![CDATA[PostgreSQL]]></category><category><![CDATA[Node.js]]></category><category><![CDATA[TypeScript]]></category><category><![CDATA[Databases]]></category><category><![CDATA[backend]]></category><dc:creator><![CDATA[Mohamed Aboelmagd]]></dc:creator><pubDate>Tue, 25 Aug 2026 18:55:02 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6a8ddff2ed1d6c279f5fab6b/98b94cb3-c919-4425-8a6c-e7fd89dd6254.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>The PostgreSQL manual is unusually direct about this:</p>
<blockquote>
<p>When an application receives this error message, it should abort the current transaction and <strong>retry the whole transaction from the beginning.</strong></p>
</blockquote>
<p>"The whole transaction" is doing a lot of work in that sentence, and it is the part that gets dropped.</p>
<p>TypeORM issue <a href="https://github.com/typeorm/typeorm/issues/9806">#9806</a> — <em>"Auto Retry options on error in transactions (e.g. Deadlock)"</em> — has been open since February 2023. Thirty 👍, six comments, no implementation. Meanwhile <code>typeorm-transactional</code>, at 188,000 downloads a week, ships <code>@Transactional()</code> with isolation levels and seven propagation modes and no retry at all.</p>
<p>So the ecosystem's actual answer to "how do I use <code>SERIALIZABLE</code> in Node" is: don't. Use <code>READ COMMITTED</code>, don't think about write skew, and hope.</p>
<p>I spent a while building the thing that issue asks for. The short version of what I found: <strong>the feature as literally requested cannot be built correctly</strong>, and the reason is more interesting than the feature.</p>
<h2>The implementation everyone reaches for first</h2>
<p>Wrap the query. It's the obvious move — the error came from a query, so retry the query:</p>
<pre><code class="language-ts">async function withRetry&lt;T&gt;(fn: () =&gt; Promise&lt;T&gt;, attempts = 3): Promise&lt;T&gt; {
  for (let i = 1; ; i++) {
    try {
      return await fn();
    } catch (e) {
      if (i &gt;= attempts || !isSerializationFailure(e)) throw e;
      await sleep(50 * i);
    }
  }
}

await dataSource.transaction('SERIALIZABLE', async (em) =&gt; {
  const from = await em.findOneOrFail(Account, { where: { id: fromId } });
  const to   = await em.findOneOrFail(Account, { where: { id: toId } });

  await withRetry(() =&gt; em.decrement(Account, { id: fromId }, 'balance', amt));  // ← here
  await withRetry(() =&gt; em.increment(Account, { id: toId },   'balance', amt));  // ← and here
});
</code></pre>
<p>This does nothing. Worse than nothing — it turns one clear error into a confusing one.</p>
<p>When PostgreSQL raises <code>40001</code>, it does not fail <em>that statement</em>. It aborts <strong>the entire transaction</strong>. The connection is now in a failed transaction state, and every subsequent statement on it — including the retry you just issued — comes back as:</p>
<pre><code class="language-plaintext">25P02  current transaction is aborted, commands ignored until end of transaction block
</code></pre>
<p>So <code>withRetry</code> burns its three attempts on a statement that is guaranteed to fail three times, then throws <code>25P02</code> instead of <code>40001</code>. You've replaced the actionable error with a meaningless one and added 150ms of sleeping to do it.</p>
<p>The same is true of deadlocks. A <code>40P01</code> victim's <em>transaction</em> is dead, not its last statement.</p>
<p>By the time you have an error to react to, <strong>there is no query left to re-issue.</strong></p>
<h2>It's worse than that: you don't know where it will fire</h2>
<p>I wrote a test asserting that under <code>SERIALIZABLE</code>, the failure surfaces at <code>COMMIT</code> — because that's how I understood SSI to work. Serializable Snapshot Isolation tracks read/write dependencies and looks for a dangerous structure; I assumed the check happened at commit time.</p>
<p>The test passed on PostgreSQL 14. Passed on 16. Passed on 17.</p>
<p>Failed on 15.</p>
<p>My first instinct was that 15 had changed something. It hadn't — and the real answer is worse for anyone writing this code.</p>
<p>I ran the write-skew reproduction 50 times against each version, no ORM in the path, recording which statement raised the error:</p>
<table>
<thead>
<tr>
<th>Version</th>
<th>Reported at <code>UPDATE</code></th>
<th>Reported at <code>COMMIT</code></th>
</tr>
</thead>
<tbody><tr>
<td>14.23</td>
<td>0</td>
<td>50</td>
</tr>
<tr>
<td>15.19</td>
<td>3</td>
<td>47</td>
</tr>
<tr>
<td>16.15</td>
<td>1</td>
<td>49</td>
</tr>
<tr>
<td>17.10</td>
<td>1</td>
<td>49</td>
</tr>
</tbody></table>
<p>15 is not an outlier — it sits between its neighbours, and the spread is noise. An earlier 25-round pass against 14.23 reported at the <code>UPDATE</code> twice; the 50-round pass above reported it zero times. <strong>Same version, same machine, same script.</strong> So the version cannot be the explanatory variable. It's a race, and how often you lose it tracks how busy the box is.</p>
<p>My test wasn't catching a quirk of PostgreSQL 15. It was flaky on every version I ran it against, and 15 was just where the coin first landed tails.</p>
<p>PostgreSQL raises <code>40001</code> as soon as its machinery <em>notices</em> the dangerous structure. Usually that's at <code>COMMIT</code>, after every statement in your transaction has already returned successfully. A few percent of the time it's the conflicting statement itself. It is not something you can predict, and it is not something you should write code against.</p>
<p>The test that replaced it asserts the only thing that's actually stable:</p>
<pre><code class="language-ts">/**
 * Where the failure surfaces is **not fixed**, and that is the point.
 * [...] Either way the *entire* transaction is aborted, so there is no single
 * statement a caller could usefully re-issue. That is what makes
 * whole-transaction retry the only sound design, and per-statement retry —
 * what TypeORM issue #9806 literally asked for — unimplementable.
 */
it('surfaces at COMMIT or at the conflicting statement, never predictably', async () =&gt; {
  const [a, b] = await produceWriteSkew(dataSource);
  const failure = reasonOf(a) ?? reasonOf(b);
  expect(failure.query).toMatch(/COMMIT|UPDATE doctor/i);
});
</code></pre>
<p>This is the general lesson, and it's the one I'd keep even if you never touch <code>SERIALIZABLE</code>: <strong>assertions about database behaviour that you derived by reasoning are hypotheses.</strong> Run them against the versions you actually support. I had a clean mental model of SSI and it was wrong in a way that only a version matrix could show me.</p>
<h2>Which pushes retry up to the transaction boundary</h2>
<p>If you can't retry the statement, retry the thing that owns the transaction. Whatever called <code>dataSource.transaction()</code> has to roll back, open a <em>fresh</em> connection and transaction, and re-run the entire callback from the top.</p>
<p>That's a small change in where the loop goes and an enormous change in what the API means, because now <strong>your callback runs more than once.</strong></p>
<pre><code class="language-ts">// ✗ BROKEN — sends two emails and charges the card twice when it retries once
@Transactional({ isolation: 'SERIALIZABLE', retry: { maxAttempts: 5 } })
async transfer(from: string, to: string, amount: number) {
  await this.accounts.decrement({ id: from }, 'balance', amount);
  await this.accounts.increment({ id: to }, 'balance', amount);

  await this.mailer.send(to, 'You received a payment');
  await this.stripe.charges.create({ amount });
  await this.kafka.publish('transfer.completed', { from, to });
}
</code></pre>
<p><code>ROLLBACK</code> undoes the two database writes. It does not un-send the email, un-charge the card, or un-publish the message.</p>
<p>The fix is to defer everything non-transactional until after the transaction is durable:</p>
<pre><code class="language-ts">// ✓ FIXED
@Transactional({ isolation: 'SERIALIZABLE', retry: { maxAttempts: 5 } })
async transfer(from: string, to: string, amount: number) {
  await this.accounts.decrement({ id: from }, 'balance', amount);
  await this.accounts.increment({ id: to }, 'balance', amount);

  runOnCommit(async () =&gt; {
    await this.mailer.send(to, 'You received a payment');
    await this.kafka.publish('transfer.completed', { from, to });
  });
}
</code></pre>
<p>Rule of thumb: <strong>if undoing it needs more than</strong> <code>ROLLBACK</code><strong>, it belongs in a commit hook.</strong></p>
<p>I suspect this constraint is a large part of why #9806 has stayed open for three and a half years. Adding a <code>retry: 3</code> option is an afternoon. Adding a <code>retry: 3</code> option that doesn't silently double-charge people requires a commit-hook mechanism, a per-attempt reset of that registry, documentation of the hazard, and a decision about what to do with in-memory state that mutated on attempt one. It's a feature that drags a design in behind it.</p>
<h2>Jitter is not a nice-to-have</h2>
<p>Two transactions that just deadlocked are <strong>synchronised by construction.</strong> PostgreSQL killed one of them at the exact instant it let the other proceed. They are phase-locked.</p>
<p>Back both off by the same <code>base · 2ⁿ</code> and they wake up together and deadlock again. And again. Exponential backoff without jitter, applied to deadlock partners, is a machine for reproducing the deadlock you just recovered from.</p>
<p>Full jitter — <code>random(0, min(cap, base · 2ⁿ))</code> — is what breaks the lock. It's the AWS architecture blog's recommendation and it's the right default here for a reason specific to this problem, not just as general good hygiene.</p>
<h2>What it actually costs</h2>
<p>Here is the part that most "add retry to your ORM" posts leave out. I benchmarked it: 600 contended read-then-write transfers per configuration, containerised PostgreSQL 17, three strategies, four concurrency levels, both a pathological and a realistic contention profile.</p>
<p>100 concurrent workers over 1,000 accounts:</p>
<table>
<thead>
<tr>
<th></th>
<th>failure rate</th>
<th>throughput</th>
<th>p99</th>
</tr>
</thead>
<tbody><tr>
<td><code>SERIALIZABLE</code>, no retry</td>
<td><strong>87%</strong></td>
<td>109 ops/s</td>
<td>168 ms</td>
</tr>
<tr>
<td><code>SERIALIZABLE</code> + retry</td>
<td><strong>0%</strong></td>
<td>109.8 ops/s</td>
<td>3,794 ms</td>
</tr>
<tr>
<td><code>READ COMMITTED</code> + ordered <code>FOR UPDATE</code></td>
<td>0%</td>
<td>382 ops/s</td>
<td>619 ms</td>
</tr>
</tbody></table>
<p>Three things worth saying plainly.</p>
<p><strong>Retry does what it claims.</strong> An 87% failure rate becomes zero. That is the entire difference between <code>SERIALIZABLE</code> being a thing you read about and a thing you can deploy.</p>
<p><strong>That p99 is a trap.</strong> Unretried <code>SERIALIZABLE</code> looks <em>twenty times better</em> on p99 — 168ms against 3,794ms. It isn't. It's fast because failing is fast: 87% of those transactions did no useful work and returned quickly. You are measuring the latency of giving up. Whenever a config looks dramatically better on latency, check what fraction of its requests succeeded.</p>
<p><strong>Retry does not make anything fast.</strong> It makes correctness <em>available</em>. Where I could enumerate the rows a transaction touches, <code>READ COMMITTED</code> with consistently-ordered <code>FOR UPDATE</code> beat <code>SERIALIZABLE</code> + retry at every single concurrency level I measured — 3.5× the throughput at this one. If you can order your locks, order your locks. Retry is for the case where you can't know in advance which rows you'll touch.</p>
<p>Under pathological contention (10 accounts, 100 workers) retry's throughput <em>falls</em> as workers are added — 77.6 ops/s at concurrency 1 down to 6.6 — because every conflict throws away a whole transaction's work. A high retry rate means retry is treating a symptom. Fix the contention.</p>
<h2>And one thing I got wrong in public</h2>
<p>I shipped <code>0.1.0</code> with observability callbacks typed like this:</p>
<pre><code class="language-ts">export type RetryCallback = (info: RetryInfo) =&gt; void;
</code></pre>
<p>Every call site was wrapped in <code>try</code>/<code>catch</code>, because an exception from someone's metrics code must never break the transaction it's measuring. I thought that was covered.</p>
<p>It wasn't. TypeScript permits an <code>async</code> function anywhere a void-returning one is expected — this compiles clean under <code>--strict</code>:</p>
<pre><code class="language-ts">onRetry: async (info) =&gt; { await metrics.push(info); }   // no error. none.
</code></pre>
<p><code>try</code>/<code>catch</code> never sees that rejection. It becomes an unhandled rejection, and Node 15+ <strong>terminates the process</strong> on those. Mid-retry, with a transaction open. A flaky metrics backend could take down the service measuring it.</p>
<p>That's <code>0.1.1</code>. Then I swept for the pattern instead of waiting for someone to hit it again — and found the diagnostic handler had the same shape, which was worse, because the diagnostic handler is the channel every <em>other</em> failure gets reported through. That's <code>0.1.2</code>.</p>
<p>If you write callback APIs in TypeScript: <code>tsc</code> will not catch this. Type-aware ESLint's <code>@typescript-eslint/no-misused-promises</code> will, at the <em>caller's</em> site — but you can't rely on your users having it enabled. Guard the call.</p>
<h2>The library</h2>
<p><a href="https://github.com/mohamedaboelmagd/typeorm-resilient-transactional"><strong>typeorm-resilient-transactional</strong></a> — <code>@Transactional()</code> for NestJS + TypeORM with SQLSTATE-classified retry, commit/rollback hooks, ordered-locking helpers, zero runtime dependencies, published with provenance.</p>
<p>It's API-compatible with <code>typeorm-transactional</code>, so migrating is one import line:</p>
<pre><code class="language-diff">- import { Transactional, runOnTransactionCommit } from 'typeorm-transactional';
+ import { Transactional, runOnTransactionCommit } from 'typeorm-resilient-transactional';
</code></pre>
<pre><code class="language-bash">npm i typeorm-resilient-transactional
</code></pre>
<p>The benchmarks above are reproducible with <code>pnpm bench</code> — the results file and the chart in it are both generated from the same run, so they can't drift from each other. If a number is in the README, it was measured.</p>
<p>I'd genuinely like to contribute the classifier and the backoff strategies upstream to TypeORM if there's appetite; I've said so on #9806. Until then, this exists.</p>
]]></content:encoded></item></channel></rss>