s3z: a hand-rolled S3 client in Rust that outruns aws-cli

S3, AWS’s object storage, is proprietary, but the wire protocol is just HTTP plus SigV4 (AWS’s request signing) plus a smattering of XML, and at some point I asked myself, in the cocky way only a Rust programmer can be cocky, how hard it could really be to build a high-performance S3 client from scratch. Beating the official client, aws-cli, isn’t the hard part, it’s a Python behemoth, so picking a compiled language gets you most of the way there for free. Getting level with mc and s5cmd, both written in Go, is another story.

The protocol turned out to be the easy part, and everything else (connection pools, multipart races, signing on retry, the syscall pattern your kernel actually likes), well, not really. s3z is what I wrote trying to combat that, a Rust S3 client and library with Python and Node bindings that ended up level with mc on downloads, ahead of mc, s5cmd and aws-cli on uploads, and lighter on memory than all three. This post goes through the walls I hit on the way there, a fair few of which I’d built myself.

The boring parts first

The first thing I started with had no networking at all, just types, config, credentials and errors, because I wanted to get the shape of the API nailed down before a single byte hit a socket.

The errors had to be typed. anyhow is great in a CLI, but people using a library want to match on what went wrong rather than read a string. Then came the HTTP layer: percent-encoding object keys (S3 is very particular about this), SigV4 signing, XML parsing and a retry loop. Going through all of it was tedious, a lot of whack-a-mole.

The final part was the transfer engine, a CLI so anyone could type s3z up ./dir and have it just work, bindings for Python and Node, and a proper benchmark. It was important to me that s3z had the dumbest API possible. I wasn’t planning for the most features, just the ones most essential to me.

The transfer engine

Broadly, s3z can be divided into a couple of core components:

• authentication
• the HTTP handler
• the interfaces for operations like download and upload
• the transfer engine, which orchestrates everything efficiently

A naive S3 uploader looks like this:

for file in files {
    client.put_object(file).await?;
}

That works, but against a remote bucket it is close to useless, because you pay a full round trip per file, and if the file is huge you risk timeouts. Chunking becomes the obvious answer once you think about these.

A scheduler plans the multipart upload up front, taking a file size in and producing part offsets and sizes, then pushes those parts into a bounded channel that a fixed pool of workers pulls from at the other end. The channel is bounded so the scheduler can’t race ahead of the network and eat all your memory, and the pool is fixed so there’s a hard ceiling on how many requests are in flight at once.

The pool is really just one loop. It keeps track of the uploads that are running, each one taking a slot. When a slot is free it takes the next part off the channel and starts uploading it, and when an upload finishes its slot opens up again. It is done when the channel is closed and nothing is still running.

Every multipart upload is wrapped in an AbortGuard, so if anything panics, errors or gets dropped halfway, the guard tells S3 to abort the upload on the way down. Without that, S3 keeps charging you for orphaned parts.

Dropped parts and stale signatures

The first real bug showed up as soon as the benchmark pushed real load: uploads would “complete” with chunks missing, and S3 would answer with 400 InvalidPart.

When an upload finished, the worker loop took its next job from the channel with try_recv, which doesn’t wait. If the channel happened to be empty at that instant, the loop carried on with one fewer upload in flight, and once the in-flight set had drained, it decided the work was over and returned, with parts still on their way down the channel.

The fix was to stop polling and use tokio::select! to wait on “a job arrived” and “an upload finished” at the same time, so the loop only ends when the channel is closed and nothing is left in flight. Simplified, the change looked like this:

loop {
    let Some(done) = tasks.join_next().await else { break };
    record(done);
    if let Ok(job) = rx.try_recv() {
        spawn(job);
    }
    tokio::select! {
        Some(done) = tasks.join_next() => record(done),
        Some(job) = rx.recv() => spawn(job),
        else => break,
    }
}

While I was in there I found a second bug. A SigV4 signature includes a timestamp, and my retry loop was resending the same signed request on every attempt, so the timestamp got older with each backoff. S3 tolerates about 15 minutes of skew, so it would have taken a big part on a slow link and a few retries to get there, but once a request is that old, S3 answers RequestTimeTooSkewed, and no number of further retries can fix that when each one carries the same stale signature.

Pre-warming that slowed startup

It’s easy to assume that more concurrency means better performance, but that’s not always true.

I configured the client to open 256 connections immediately when it was created. The idea was to have all the connections ready before the first uploads started, so the uploads wouldn’t have to wait for TLS handshakes. This seemed like a good optimization at first, so I made the constructor async and moved on.

But opening all those connections upfront added 250 ms to 1 second to every run. For a CLI that might be run hundreds of times, that startup cost adds up quickly. It also made the benchmark results inconsistent. Some runs finished the connection setup quickly, while others got stuck waiting for slower TLS handshakes. As a result, the benchmark had 2 groups of results instead of one consistent result.

Then I realized that reqwest already has a connection pool that creates connections when they’re needed. A new connection only took around 2 ms, which is tiny compared with the time spent waiting for an S3 request. So I removed my connection pre-warming.

The result was pretty much exactly what I wanted:

• Startup became faster.
• The benchmark results became more consistent.
• Throughput stayed the same.
• The constructor could be synchronous again.

1.5 GiB of RAM for 40 MB files

After dealing with slow startups, it was time for memory to teach me a lesson. Files under the multipart threshold (50 MiB at the time) took a simple path: read the whole file into memory and send it. Well, with 32 workers against a remote endpoint, where each upload sits in flight for hundreds of milliseconds, that added up to a peak of 1,566 MiB, which is a lot of memory to spend on uploading some 40 MB files.

I already knew the solution to this: lower the threshold and start streaming the file from disk in 256 KiB chunks, the same way the multipart path already did. This brought it down to 85 MiB with no loss in throughput.

Network shenanigans

There are some bugs and optimizations that take a wizard to catch by reading the code, and I never went to Hogwarts. These ones I only found through my extensive, albeit ad hoc, testing, by running the real workload and watching what the kernel did with it.

Slots that sat empty

My first version of the select! fix from earlier waited for the next part before it checked for finished uploads, so a finished upload kept its slot until a new part showed up. Now it waits for both at once:

tokio::select! {
    // an upload finished, its slot is free
    Some(done) = tasks.join_next() => record(done),
    // a part arrived and there is a free slot
    job = rx.recv(), if tasks.len() < slots => spawn(job),
}

Everyone retrying together

When S3 says “slow down”, you wait and try again. All my workers waited the same amount, so they all came back together and got throttled again. A random wait, called jitter, spreads them out:

sleep(delay).await;
sleep(delay / 2 + random(0..=delay / 2)).await;

40 ms on every write

Every chunk I wrote to the socket stalled for about 40 ms. It was 2 TCP features waiting on each other, Nagle’s algorithm on the sender and delayed ACKs on the receiver. Turning Nagle’s algorithm off with tcp_nodelay sends the data right away.

Asking which HTTP to speak

Every new connection first agreed on an HTTP version with the server. S3 only speaks HTTP/1.1, so there was nothing to agree on. Forcing HTTP/1.1 skips the question.

One writer thread per file

Downloads had a problem of their own. A big download comes down as several parts at once, and each part writes its chunks straight into its own place in the file. That is what pwrite does, it writes at a position you give it.

Writing to disk blocks, and async code shouldn’t block, so tokio gives you spawn_blocking, which runs the work on a separate pool of threads. My first version did that for every single write:

// for every chunk of every part
spawn_blocking(move || file.write_all_at(&chunk, offset)).await?;

That works, but for a 1 GB file it’s thousands of trips through a thread pool that everything else in your program is sharing too. The OS also doesn’t write to disk right away, it keeps the data in memory (the page cache) and flushes it later. When it decided to flush under all that write pressure, everything piled up behind it, and the same 5 GB download on the same network took anywhere from 4 to 29 seconds.

The fix was to give each file one thread that does nothing but write. The parts drop their chunks on a queue, and the thread writes them out one after another:

std::thread::spawn(move || {
    while let Some(chunk) = queue.blocking_recv() {
        file.write_all_at(&chunk.buf, chunk.offset)?;
    }
    file.sync_all()
});

No more trip through the pool for each write, and nothing to fight over. The same 5 GB download now takes 4 to 6 seconds every time. The queue has a fixed size, so if the disk falls behind the downloads wait, instead of chunks piling up in memory.

Two smaller wins came out of the same area:

• Fewer writes. Each part now collects 512 KiB before sending it to the writer, which cut the writes from about 64,000 per GB to about 2,000.
• Fewer parts. I had capped download parts at 256 MB, which S3 never asked for. Without the cap, a 10 GB file at concurrency 8 comes down as eight 1.25 GB streams, not forty 256 MB ones, so far fewer requests to sign and wait on.

Choosing part size and concurrency

This is the part which had a lot of back and forth. At first every part was a fixed 50 MiB, which is fine for some files and bad for most:

There is no defined formula for this, it’s a problem of finding the right heuristic, and it was a lot of trial and error. For uploads I now aim for about twice as many parts as streams, because PUT latency varies and a bit of queue means a free stream always has the next part ready.

For downloads I aim for one part per stream, because the server starts sending as soon as you ask and fewer Range requests win. The part size also grows with the file now, so the 10,000 limit is only in reach past about 2.4 TiB, and there s3z panics with a clear message, it doesn’t start an upload it can’t finish.

Then I noticed the number of streams shouldn’t be fixed either. A 128 MB file with 8 streams just allocates 8 buffers and barely uses them. Files up to 256 MiB now get 2 streams, up to 2 GiB get 4 and anything bigger gets 8. That dropped memory for the small-file workload from between 79 and 92 MB to between 30 and 39 MB, while big files still saturate the link.

Python and Node bindings

Creating Python and Node bindings was mainly an experiment to see if the library could work across multiple ecosystems while keeping one shared Rust core. The main challenge was that some Rust types borrowed data, which doesn’t translate cleanly to Python or JavaScript. So I changed those types to own the data they needed, allowing the same API to work across Rust, Python and Node.

Node also had a runtime issue, because NAPI-RS already manages a tokio runtime, so I had to reuse the existing runtime instead of creating another one. The result was one Rust implementation that could be used from Rust, Python and Node without maintaining separate implementations.

You can make some backends happy some of the time

I’d set up 4 S3-compatible backends locally (MinIO, RustFS, SeaweedFS and Garage) so I could catch my own protocol assumptions before users did, Hyrum’s Law and all that. RustFS rejected every upload with UnexpectedContent, which took me far too long to trace to its wanting an explicit Content-Length header on every signed request, even where reqwest would have filled one in later anyway.

I’d love to tell you all four stayed in the benchmark, but RustFS started hanging at random under load, and after half a day of trying to pin down why, I took it out rather than publish numbers from a backend that might hang in the middle of a run.

A benchmark I can trust

I did not expect to spend this much time on the benchmark, but the whole point of s3z is “this is faster”, so the benchmark has to be something I trust. Every run gets a warmup, noisy runs get more samples until the numbers settle, and a compare command fails a change only if the difference is statistically real and bigger than the noise I’ve measured.

Even so, it almost lied to me once. mc was losing badly against Garage and I nearly published a chart that said so, until I worked out that mc wasn’t picking up the region from the environment the way the other tools were. The bug was in my setup.

Results

The reference run is committed in the repo: three 256 MB files, 10 to 30 runs per tool, against MinIO, SeaweedFS and Garage running in Docker on my laptop. On downloads s3z is level with mc and ahead of s5cmd and aws-cli on all 3 backends, and on uploads it’s ahead of all 3 tools everywhere. Downloads come first, with MinIO shown and the tabs switching to the other backends:

Uploads are where the transfer engine and all the fixes above pay off, with s3z ahead of mc by about a quarter of a second on MinIO and by well over a second on Garage:

The part I’m proudest of, though, is how little memory it needs to get there:

The 3 fixes from this post that moved the numbers most:

So much to break, so much to gain

Tinkering with s3z humbled me. I realized how deep the ocean is, and my love for coral beads. I know they say cache invalidation and naming are the hardest things in computer science, but maybe they never had to deal with concurrency and contention. Either way, I am glad for all the pitfalls, and I am excited for all the things I’m yet to break.

Try it

The code is at jaeaeich/s3z↗. The CLI installs with one curl, the library is on crates.io, the Python wheels are on PyPI and the Node package is @jae_aeich/s3z on npm. There are also Axum, FastAPI and Elysia examples that all expose the same API, using s3z directly from Rust or through the bindings.