#1051

Using Supabase transaction pooler, a serializable transaction can silently discard all its writes

Openbrianmgrayopened 7mo ago

Bug report

I can't find an existing bug report for this, am actually quite amazed by that.

Describe the bug

When using the Supabase transaction pooler (PgBouncer, port 6543) with serializable isolation, a transaction with serializable isolation level can silently discard all its writes. No error is thrown, no indication of failure, and returns a fully populated row, but NO rows are persisted to the database.

To Reproduce

I built this repo with a minimal reproduction of the bug. It is the same essential test I wrote to prove the bug in our codebase: https://github.com/brianmgray/supabase-pool-bug-repro

Instructions in the README.md

You will need to connect it to a Supabase project, and provide the session pool URL and the transaction pool URL in a .env file, then run:

corepack enable
pnpm install
pnpm db:migrate
pnpm test:session # should fail
pnpm test:transaction # should succeed

Expected behavior

A database transaction should have exactly two outcomes: commit (data persists, no error) or abort (error thrown, no data written).

With the Supabase transaction pooler under serializable isolation, a third unexpected outcome occurs: the transaction appears to commit (no error, RETURNING returns data) but all writes are silently discarded. This should never be possible.

Screenshots

Sample output:

❯ pnpm test:session

> supabase-pool-bug-repro@1.0.0 test:session /Users/brian/workspaces/fuguUX/archive/supabase-pool-bug-repro
> POOL=session node repro.js


--- Session Pooler (port 5432) ---

=== Phase 1: READ COMMITTED allows overdraft ===

  ℹ️  Only 1 hold succeeded — transactions serialized by timing

=== Phase 2: SERIALIZABLE + INSERT...RETURNING ghost rows ===

  Serialization failures: 6 (expected — SSI working)
  Apparent commits (RETURNING returned data): 2
  ✅ All 2 committed rows verified — no bugs found
❯ pnpm test:transaction

> supabase-pool-bug-repro@1.0.0 test:transaction /Users/brian/workspaces/fuguUX/archive/supabase-pool-bug-repro
> POOL=transaction node repro.js


--- Transaction Pooler (port 6543) ---

=== Phase 1: READ COMMITTED allows overdraft ===

  ℹ️  Only 1 hold succeeded — transactions serialized by timing

=== Phase 2: SERIALIZABLE + INSERT...RETURNING ghost rows ===

  🐛 BUG: INSERT...RETURNING returned idempotency_key=hold-ssi-3-af7aa9f6-2f35-46c7-b67e-bb6a8a1cbdd6 but the row is NOT FOUND after commit
  Serialization failures: 5 (expected — SSI working)
  Apparent commits (RETURNING returned data): 3
  🐛 1 of 3 rows NOT FOUND after commit  ← the bug

System information

  • OS: MacOS
  • Browser (if applies): n/a
  • Version of supabase-js: n/a, using postgres and in our app drizzle (same result)
  • Version of Node.js: v22.22.0

Additional context

None

Activity

6
kashaffatimajaffrey-design4mo ago

Reproduced on June 25, 2026 against a fresh Supabase project (Seoul region, Nano compute, PostgreSQL 15) using the original repro repo. Session pooler (port 5432):

Serialization failures: 6 (expected - SSI working) Apparent commits: 2 ✅ All 2 committed rows verified - no bugs found Transaction pooler (port 6543): 🐛 BUG: INSERT...RETURNING returned idempotency_key=hold-ssi-3-f7256913-1ef5-4831-90b2-4e8419f0481e but the row is NOT FOUND after commit Serialization failures: 5 (expected - SSI working) Apparent commits: 3 🐛 1 of 3 rows NOT FOUND after commit

Root cause hypothesis: Supavisor's transaction pooler appears to release the server connection back to the pool before the serializable transaction's conflict detection has fully resolved. Postgres SSI uses deferred conflict checking, it may silently roll back the transaction after Supavisor has already forwarded the RETURNING result to the client, making it appear as a successful commit. This violates the guarantee that a transaction has exactly two outcomes: commit or abort.

Happy to dig further into the Supavisor connection lifecycle if that's helpful, is there a preferred entry point in the codebase?

kashaffatimajaffrey-design4mo ago

Reproduced on June 25, 2026 against a fresh Supabase project (Seoul region, Nano compute, PostgreSQL 15) using the original repro repo. Session pooler (port 5432):

Serialization failures: 6 (expected - SSI working) Apparent commits: 2 ✅ All 2 committed rows verified - no bugs found Transaction pooler (port 6543): 🐛 BUG: INSERT...RETURNING returned idempotency_key=hold-ssi-3-f7256913-1ef5-4831-90b2-4e8419f0481e but the row is NOT FOUND after commit Serialization failures: 5 (expected - SSI working) Apparent commits: 3 🐛 1 of 3 rows NOT FOUND after commit

Root cause hypothesis: Supavisor's transaction pooler appears to release the server connection back to the pool before the serializable transaction's conflict detection has fully resolved. Postgres SSI uses deferred conflict checking, it may silently roll back the transaction after Supavisor has already forwarded the RETURNING result to the client, making it appear as a successful commit. This violates the guarantee that a transaction has exactly two outcomes: commit or abort.

Happy to dig further into the Supavisor connection lifecycle if that's helpful, is there a preferred entry point in the codebase?

Snehil-Shah4mo ago

@v0idpwn can I help debug this? If the team is not on it yet?

Snehil-Shah4mo ago

@v0idpwn can I help debug this? If the team is not on it yet?

kashaffatimajaffrey-design2mo ago

Following up on my June reproduction with a narrower result: the ghost row only appears when the client uses prepared statements. That moves the likely cause away from SSI conflict-detection timing, which is what I guessed originally, and onto the extended-protocol path in transaction mode.

Re-tested today against a Supabase project on PostgreSQL 17.6 (Seoul, aws-1-ap-northeast-2, Nano), using the original repro repo unchanged.

  1. Still reproduces on PG 17.6, my earlier report was PG 15.

transaction pooler, postgres.js defaults: 10/16 runs produced a ghost row

  1. It does not reproduce with prepare: false. Same project, same pooler, same repro, one option changed:

const sql = postgres(url, { max: 10, prepare: false }); prepare: true (default): 10/16 runs produced a ghost row (42 apparent commits) prepare: false : 0/16 runs produced a ghost row (32 apparent commits)

(Two independent batches, 10 runs then 6, same result each time.)

The apparent-commit totals are the clearest part. The prepared path reports exactly 10 more "successful" commits than the unprepared path across the same number of runs, and there were exactly 10 ghosts. The extra commits are the ghosts; nothing else differs between the two arms.

  1. It does not reproduce through psycopg2. I wrote an equivalent probe in Python, 8 concurrent SERIALIZABLE transactions, same table shape, same pooler, and ran it 5 times. 40 transactions, zero ghosts: every one either committed and persisted, or failed cleanly with 40001. psycopg2 does not use named prepared statements by default, which fits (2).

  2. Adding two round-trips inside the transaction suppresses it. I instrumented each transaction with pg_backend_pid() and txid_current() immediately after BEGIN and again before COMMIT, to test whether the transaction was being split across server backends. With that instrumentation the bug stopped reproducing, 0/12 runs, so I could not capture a ghost with the PIDs attached. The instrumentation itself is the confound: two extra round-trips are enough to close the window.

Taken together the trigger looks like prepared statements plus a short pipelined transaction (BEGIN / SELECT / INSERT ... RETURNING / COMMIT with no intervening round-trips), rather than anything specific to SSI. SERIALIZABLE may just be what makes the failure visible, by producing the aborts that then go unreported.

I could not narrow it further from the client side, distinguishing "the COMMIT's error response was dropped" from "the COMMIT reached a different server connection" needs Supavisor-side visibility I do not have. If someone with access can correlate a ghost transaction against the pooler's server-connection assignment and the backend's actual commit/abort, that should settle it.

In the meantime, prepare: false is a usable workaround for postgres.js users on the transaction pooler, 0/16 here, though I would not call 16 runs proof of absence.

Scripts are in a PR against the repro repo: brianmgray/supabase-pool-bug-repro#1, a prepare: false variant of repro.js, the PID/txid probe, and the psycopg2 control, with the full numbers in probes/README.md. Happy to run further experiments against this project if a maintainer wants a specific one.

cekuu352mo ago

The prepared-statement correlation points at a mechanism worth testing explicitly: with named statements in play, transaction-mode pooling has to route by statement identity, which makes it possible for the short pipelined sequence BEGIN / SELECT / INSERT ... RETURNING / COMMIT to be split across backends - BEGIN executing on backend A, the later steps on backend B. That produces exactly your symptom without requiring any error response to be dropped: backend B receives COMMIT while holding no open transaction, answers with a successful COMMIT tag (emitting only a "there is no transaction in progress" warning, which drivers such as postgres.js do not surface), and backend A's abandoned transaction is rolled back when its connection is cleaned up. The RETURNING row you saw was real - it was produced before the split. This also explains why prepare: false eliminates it: the simple-protocol path gives the pooler nothing to pin routing decisions on.

Two discriminators that need neither Supavisor internals nor extra client round-trips (your pid-probe's extra round-trips closing the window is itself informative):

  1. Use the project's Postgres logs instead of wire taps. Dashboard -> Logs -> Postgres carries per-session lines with backend PIDs. Correlate the ghost run's BEGIN, INSERT and COMMIT lines: a COMMIT under a different PID than BEGIN confirms the split directly. If instead backend A shows the serialization-failure message and no COMMIT ever arrives, you are in the dropped-error-response case.

  2. Check pg_stat_activity immediately after a ghost run (SQL editor or session pooler): sessions sitting in idle in transaction with backend_xid set near the ghost timestamp are the fingerprint of the abandoned-open-transaction path. Their absence points back toward a swallowed error.

One framing note on the June hypothesis: PostgreSQL cannot roll a user transaction back after returning a successful COMMIT tag, so "the connection was released before conflict detection resolved" has to reduce to one of these two cases anyway - either the COMMIT never reached the backend that owned the transaction, or its failure response was lost between backend and client.

Practical notes meanwhile: your prepare: false result doubles as the immediate mitigation for postgres.js users, and independently of this specific bug, SERIALIZABLE over transaction-mode pooling remains structurally fragile because SSI state is per-backend - serializable write workloads are safer on the session pooler or a direct connection, with REPEATABLE READ plus explicit retry as the usual production compromise.