100,000 Lines of C Later: pgSafe at PGDay UK 2026

Yesterday I had the pleasure of speaking at PGDay UK 2026 in London, at the Cavendish Conference Centre. The talk was called 100,000 Lines of C Later: Re-architecting Enterprise Postgres Backups in Go. It was about the design of a Postgres backup tool: the rules such a tool has to obey, and what those rules look like when you rebuild the tool from scratch in Go, which is what pgSafe is. The slides are available here.

It was very well received, the feedback afterwards was positive, and people were intrigued by the possibilities. For those who weren't there, here is what I talked about.

What happened in April 2026

pgBackRest is one of the two de facto enterprise backup tools for Postgres, with its first stable release dating back to 2016. It started life as Perl and is now written in C, and it has a decade of production lessons incorporated into its code. In April 2026 it lost its corporate backing, and the maintainer archived the repository. A few weeks later, it turned out the project could survive, and it is being maintained again.

Nobody did anything wrong here: funding stopped, and somebody made a reasonable call. But for those few weeks, the question was: what do we back up with?

Why was that a problem?

pgBackRest is about 96,000 lines of dense C, and another 80,000 lines of test harness: manual memory management, custom networking, its own protocols. It is excellent code, maintained by very few people: not many can understand it well enough to work on it.

Critical infrastructure needs alternatives that more people can maintain.

So I started writing one.

Introducing pgSafe

pgSafe is written from scratch in Go. It shares pgBackRest's concepts and operational rules, and none of its code. The goal is functionality parity for the common deployments: full and incremental backups, point-in-time recovery, five storage backends (POSIX, S3, Azure Blob, GCS, SFTP), and PostgreSQL 13-18 support. I am happy to announce that the development team (just me for now) is already half the size of pgBackRest's (j/k: thank you for all your work, David and Stefan ❤️).

It is at alpha maturity. Please don't back up production with it yet. There is a gaps document in the repo listing what is missing and what was left out on purpose.

Ground rules

Some ground rules I set for myself before setting off on this experiment:

• Read pgBackRest's source to learn the rules. Write the Go from the rules, not from the C. This epitomises the best of open source: learning. Not copying or freeloading.
  • Using software for free is fully intended by open source licences but taking advantage of someone's hard work, making lots of money off of it, and giving nothing back is a completely different thing.
• Every module is a sealed box with a small public interface.
• TDD: no code without unit, integration and end-to-end tests.
• Every line of code has to justify its existence.

"With every modern tool at my disposal"

Is included in the talk's abstract. That includes AI coding assistance. So is this vibe-coded AI slop?

No, because the assistant works under the same rules as I do. KISS, DRY, UNIX-style sealed modules, stdlib first, no ORM, and non-negotiable TDD (unit, integration, end-to-end) on every change. The architecture, the tenets and the invariants are mine. Every code decision is reviewed, which makes supervision the actual job. The responsibility is mine, and that cannot be delegated.

The other modern tool is Go itself: a standard library with HTTP, TLS and crypto, JSON built in, generics, a garbage collector, and errgroup for parallel goroutine coordination. The go toolchain has go test, golangci-lint and the race detector built in. It can also perform cross-compilation and produce static binaries at the end.

What does a backup tool really do?

1. Tell Postgres a backup is starting
2. Copy every file in the data directory to somewhere else
3. Tell Postgres the backup is finished
4. Make sure the WAL from that window is safely archived
5. Write down what you copied, with checksums
6. Be able to put it all back, at a point in time of your choosing

Sounds simple, right? The hard part is the ordering. Every hard bug in backup tooling is an ordering or durability bug, not a copying bug.

The WAL bracket

pg_backup_start(), copy files, pg_backup_stop(). The files you copied in between are individually inconsistent, because Postgres kept writing to them while you were reading them. What repairs all of that is the WAL replay of every single segment from the start LSN to the stop LSN. Without that WAL, the backup is worthless. Everything else in the tool exists to serve this bracket.

pgSafe has three ways to get the WAL: archive, the default, where archive_command copies each completed segment and you get full PITR reach; stream*, which is pg_basebackup --wal-method=fetch with the WAL included; and walgrab, where our worker reads $PGDATA/pg_wal after the stop. The last two give a self-contained backup, and restore doesn't care which one you used: it looks inside the backup for pg_wal first, and for the archive source it stages the bracket segments and writes a restore_command so Postgres pulls the rest on demand and follows timeline switches by itself.

* "stream" is probably a misnomer, since pg_basebackup has its own --wal-method=stream, which is a different thing. I'll re-examine this naming decision.

The ten invariants

Every invariant is a lesson somebody learned the hard way. They are not implementation details, they don't change between releases, and each one has a name, a reason and a named test. A regression on any of those tests must block the release, and they are valid across all three operating modes.

1. Ordering: stop, wait, then manifest. Make every file durable, then call pg_backup_stop(), then wait until every required WAL segment is in the archive, and only then write the manifest.
2. The manifest is the source of truth: written last, atomically. Its presence means "this backup exists and is valid".
3. Resume on checksums, not timestamps. Postgres can change a file's contents within a single mtime tick, so resuming based on mtime can produce a silently inconsistent backup.
4. Retention cannot remove a backup that a running backup depends on.
5. Probe the WAL archive before starting. Don't bother Postgres until you know that the WAL can be be archived successfully.
6. fsync ordering on POSIX filesystems. The bytes are durable before the filename is published to the directory. On object stores, which have no rename, a conditional copy without clobbering does the same job as an atomic rename.
7. Re-pushing a WAL segment: same bytes is a no-op, different bytes is an error. Postgres can re-emit a WAL segment during crash recovery, so it makes sense to check the hash.
8. Backup from a standby only if it is actually replaying from the primary. A standby that is disconnected may result in a backup that's older than it claims to be.
9. Consistent encryption recipients across a backup, including workers. A backup whose files were encrypted for different keys cannot be wholly restored.
10. Multi-storage: backup is considered complete when at least one backend commits.

The reasoning and the tests behind each rule are in INVARIANTS.md in the repo.

How to test an invariant

"Kill it and see" needs some determinism. Every durability step gets a named injection point (StepWriteTemp, StepFsyncFile, StepRename, etc.) We inject a failure at each step and then we check what is actually on disk.

Three design tenets

Three constraints defined the architecture:

Tenet 1: Postgres never knows pgSafe exists. No extension, no shared library, no hooks. Cluster identity is determined from SQL functions (pg_control_system(), pg_control_checkpoint(), etc.), not from reading pg_control. WAL archiving goes through archive_command, not archive_library: an archive_library runs inside the backend, and a bug there can delay checkpoints or take the cluster down. The cost is 1 to 2 ms per WAL segment (16 MB): practically nothing. A backup tool must not be able to crash or block the database.

Tenet 2: Standards and libraries first. KISS, and don't reinvent the wheel. pgSafe uses Postgres's native backup_manifest format rather than a custom one, so we can use pg_verifybackup to check backups. PG17's WAL summarizer for incrementals, and pg_combinebackup to reconstruct their chain upon restoring. We use the cloud vendors' own SDKs for S3, Azure and GCS storage. age for encryption, and net/http and crypto/tls from the standard go library. Less code on our side means fewer bugs we have to own.

Tenet 3: No credentials on the database host. The database host is the host most likely to be compromised, which makes it a poor location for long-lived S3 keys. So the caller issues a narrow, short-lived credential per backup: - S3: AssumeRole limited to PutObject on one prefix - Azure: A service SAS that is write-and-create and HTTPS only - GCS: Impersonated token valid for about an hour This is delivered in memory over the worker channel, and is never written to disk.*

* One exception: archive_command runs on the database host, so archive-push still reads a storage key from the config file there. The fix is easy, and it's coming very soon.

Three operating modes

The only distinction is whether there is a pgSafe worker binary on the database host.

PG-native (no worker):

• simple: one replication connection, pg_basebackup tar stream.
• remote-parallel: n connections calling pg_read_binary_file(). As a side benefit of reading block by block, page checksums are validated on the client side as it's reading.

Both work wherever Postgres is connectable, with no shell access needed.

pgSafe mode (worker): a worker process on the Postgres host reads $PGDATA directly via OS syscalls, with n goroutines reading files in parallel, and streams the bytes directly from PG host to storage (bytes move only once, they don't pass through the caller or libpq). The worker is spawned locally as a subprocess when the caller is on the same host, or over SSH. It's the only mode with scoped credentials, and Postgres sees only one or two connections for the whole backup regardless of how many workers you run.

The caller always gets the WAL bracket over a Postgres connection. In simple and remote-parallel, the bytes come back through the caller and go to storage. In pgSafe mode, the caller employs a worker process on the database host over JSON-RPC (or over SSH if caller is on a different host), and the credentials exist only in the worker's memory. WAL always arrives through archive_command.

There is no --mode flag

Mode is a property of your deployment, not via invocation. The caller reads the config and works out the deployment shape from it. If pg.host is set and reachable over SSH, you get worker mode. If several workers are configured and there is no pg.host, you get remote-parallel. In any other case, you get simple. The only parallelism option you set is --workers n. Every run prints out the resolved topology in plain English so you can see how the bytes actually flowed during the backup.

Where did 100,000 lines go?

More than 50% of the C code had nothing to do with backups:

What's in those 51,000 lines?

Things a C project in 2011 had to write itself:

• HTTP client, TLS and socket layer
• Request signing for S3, Azure and GCS by hand: SigV4, SharedKey, JWT
• Wrappers around OpenSSL for ciphers and hashes, and its own copy of MD5 (FIPS)
• Strings, lists, variants, key-value maps, JSON, XML, a pack format
• Memory contexts, TRY/CATCH macros
• A code generator for the config system, and 11,796 generated lines of code

In Go we can use net/http, crypto/tls, encoding/json, generics, and the Garbage Collector. Additionally, the cloud vendors' own SDKs for their APIs. None of these 51,000 LoC is about backups: it's the cost of using the C language.

Several thousand more lines are absorbed by Postgres itself: PG17's WAL summarizer replaces a page-diff engine, backup_manifest replaces a private manifest format, pg_combinebackup replaces incremental reconstruction, and pg_verifybackup replaces a verify engine. Using the native manifest format means anyone can audit a pgSafe backup with tools that already come with Postgres.

Parallel file copy in Go

errgroup replaces the worker pool, in a few dozen lines. pgBackRest's protocol/ (processes, handshakes, wire protocol) is 3,256 lines. pgSafe's parallel file copy is this:

g, ctx := errgroup.WithContext(ctx)
g.SetLimit(workers)
for _, f := range files {
    g.Go(func() error { return copyOne(ctx, f) })
}
return g.Wait()

The first error cancels the context: every worker sees it and stops. Six lines, no supervision code, no wire protocol. What's left? The actual backup orchestration.

The tally so far

• pgBackRest: 96,000 lines of C, plus 80,000 of tests.
• pgSafe: 18,000 lines of Go, plus 15,000 of tests
  • Five storage backends, six Postgres versions and three operating modes.

The layers of pgSafe.

But this isn't about lines of code. It's about how many people can understand the code, and how easy it is to maintain. This is something one person could maintain in their spare time.

Caveats: pgSafe is much, much younger and does fewer things (see FUNCTIONALITY_GAPS.md), and it has not been tested against 10 years of user edge cases.

The parts that cannot shrink

The resume protocol:

• Every 10 files, checkpoint what arrived and the hashes of the stored bytes.
• The next run lists the storage and decides, per directory:
  • Manifest is present? Completed, skip.
  • No checkpoint file? There's nothing to resume.
  • Checkpoint too old? It's a stale backup attempt, delete.
  • Same version, type, parent backup, compression, keys? Resume.
• Then re-hash what is there, and delete and re-upload anything mismatched.
• backup_label and tablespace_map are never reused: fresh ones for this attempt.

The other three:

• WAL bracket guarantees: ordering, timeline tracking, standby coordination, waiting.
• Credential scoping: 4 backends, 4 different permissions systems.
• Atomic rename substitutes: 5 backends, 5 different conditional write procedures.

These are mostly contracts with things outside our process, and language choice can't affect them.

Testing (trust nothing)

Unit, integration and end-to-end tests (using real PG13 to PG18 in containers) on every change, no exceptions. Fault injection deterministically crashes the writer at every step. We don't trust status flags, exit codes, and of course not "works on my laptop". There is a local CI script to use as a commit gate, and has identical steps to the CI workflow to prevent inconsistncy.

And remember:

• Backups that have never been restored are...?

• NOT backups.

Re-architecting vs porting: is it worth it?

Here are some questions you need to ask yourself before you start such an endeavour:

• How much of the code is simply language baggage?
• Has the platform absorbed some of the complexity since the original project was written?
• Can you write down the invariants without reading the code?
• Is the test suite the real asset? You can port your understanding of it.

If some of these apply, a rewrite might be smaller than it looks. If none apply, you may be about to reproduce ten years of bugs (or spawn new ones).

AI assistance can save you typing, but it can't produce the rules for you.

What's not there yet

For completeness, here's what pgSafe doesn't do yet:

• WAL archiving is synchronous, one segment at a time (pgBackRest has asynchronous and parallel archiving)
• No delta restore
• No file bundling
• Incremental backups are PG17+ only, and only in simple mode
• The WAL archive is not yet encrypted or compressed at rest like the base backups. This is high on the priority list to address

The gaps document has the complete list, including what was left out on purpose.

Where we are, and what is needed

pgSafe is open source under the PostgreSQL Licence, and it is alpha code: Please try it, read the code, break the code, and tell me where the design is wrong.

Potential contributors will find INVARIANTS.md. I suspect there are more than ten, so let me know if you spot another.

If you have seen a backup-related incident, and it led you to a rule I don't know yet, please tell me about it!

We certainly need:

• More eyes for review 👀
• Testing against the cloud backends (can be $$$)
• Addressing the missing features

To summarise

• The hard problems in a backup tool are ordering and durability, not moving data
• Write your invariants down, name them, and test them
• Standards and libraries first
• Keep credentials off the database host
• Half of an old C codebase can be infrastructure that the language lacks rather than logic that solves the problem (cough, cough 🐘)
• Test-restore your backups

Many thanks to the PGDay UK organisers, fellow speakers, sponsors, and to everyone who came, asked questions and stayed to talk afterwards!

Links

• Slides: vyruss.org/computing/slides/pgdayuk2026_pgsafe.pdf
• Code: github.com/vyruss/pgSafe

Any feedback is welcome, including in person at Postgres events, on Mastodon at @vyruss@fosstodon.org or on Bluesky at @vyruss.org.