Streaming PostgreSQL to Parquet at Wire Speed: Zero-Copy Serialization in Rust

Most database export jobs are accidentally row-oriented. They fetch a row, allocate a set of values, convert those values into another representation, and eventually rebuild them as columns. It is convenient to write and expensive to operate.

postgres_to_parquet starts with a different premise: extraction is a bounded stream transformation. The process should move through a fixed amount of memory regardless of table size, and values should stay in their wire-compatible representation for as long as possible.

Preserve the shape of the data

PostgreSQL’s binary COPY protocol avoids text parsing on the hot path. The extractor reads batches, maps the schema once, and appends values directly to their target column writers. A row is only a transport envelope, not the abstraction that governs memory layout.

while let Some(batch) = copy_reader.next_batch().await? {
    parquet_writer.append(batch.columns())?;

    if parquet_writer.is_full() {
        parquet_writer.flush_row_group()?;
    }
}

The purpose of this loop is not cleverness. It makes the resource contract visible: batches arrive, bounded column buffers fill, a row group flushes, and the process continues.

A practical throughput model

The sustained rate is limited by the narrowest stage in the pipeline:

$$T_{pipeline}=\min(T_{postgres},T_{network},T_{encode},T_{storage})$$

Optimizing a serialization step above the network or storage ceiling will not increase end-to-end throughput. Instrumenting each stage does, however, reveal where a change is worth making. The useful metrics are bytes read, rows decoded, row groups flushed, allocation pressure, and write latency.

Zero-copy is a discipline

Zero-copy does not mean that no allocation ever happens. It means copies are deliberate and deferred until a format boundary requires them. In this path, that keeps CPU cycles available for compression and encoding rather than transient string and value objects.

The final design is simple to reason about: a schema-bound reader, a bounded batch, a column writer, and backpressure at every expensive boundary. That is what lets an export remain predictable when it moves from a test table to a production-sized one.