# s3z: a hand-rolled S3 client in Rust that outruns aws-cli > I built s3z, an S3 client in Rust, and got it level with mc on downloads and ahead of mc, s5cmd and aws-cli on uploads, mostly by undoing my own clever ideas. Javed Habib · Apr 28, 2026 · #s3 #rust #perf https://jaeaeich.com/writing/s3z S3, AWS's object storage, is proprietary, but the wire protocol is just HTTP plus SigV4 (AWS's request signing) plus a smattering of XML, and at some point I asked myself, in the cocky way only a Rust programmer can be cocky, how hard it could really be to build a high-performance S3 client from scratch. Beating the official client, `aws-cli`, isn't the hard part, it's a Python behemoth, so picking a compiled language gets you most of the way there for free. Getting level with `mc` and `s5cmd`, both written in Go, is another story. The protocol turned out to be the easy part, and everything else (connection pools, multipart races, signing on retry, the syscall pattern your kernel actually likes), well, not really. [**s3z**](https://github.com/jaeaeich/s3z) is what I wrote trying to combat that, a Rust S3 client and library with Python and Node bindings that ended up level with [`mc`](https://github.com/minio/mc) on downloads, ahead of `mc`, [`s5cmd`](https://github.com/peak/s5cmd) and [`aws-cli`](https://aws.amazon.com/cli/) on uploads, and lighter on memory than all three. This post goes through the walls I hit on the way there, a fair few of which I'd built myself. > **short on time?** > > The numbers and graphs are at the end, in the > [results](#results) section. ## The boring parts first The first thing I started with had no networking at all, just types, config, credentials and errors, because I wanted to get the shape of the API nailed down before a single byte hit a socket. The errors had to be typed. `anyhow` is great in a CLI, but people using a library want to `match` on what went wrong rather than read a string. Then came the HTTP layer: percent-encoding object keys (S3 is *very* particular about this), SigV4 signing, XML parsing and a retry loop. Going through all of it was tedious, a lot of whack-a-mole. The final part was the transfer engine, a CLI so anyone could type `s3z up ./dir` and have it just work, bindings for Python and Node, and a proper benchmark. It was important to me that s3z had the dumbest API possible. I wasn't planning for the most features, just the ones most essential to me. ## The transfer engine Broadly, s3z can be divided into a couple of core components: - authentication - the HTTP handler - the interfaces for operations like download and upload - the transfer engine, which orchestrates everything efficiently A naive S3 uploader looks like this: ```rust for file in files { client.put_object(file).await?; } ``` That works, but against a remote bucket it is close to useless, because you pay a full round trip per file, and if the file is huge you risk timeouts. Chunking becomes the obvious answer once you think about these. A scheduler plans the multipart upload up front, taking a file size in and producing part offsets and sizes, then pushes those parts into a bounded channel that a fixed pool of workers pulls from at the other end. The channel is bounded so the scheduler can't race ahead of the network and eat all your memory, and the pool is fixed so there's a hard ceiling on how many requests are in flight at once. The pool is really just one loop. It keeps track of the uploads that are running, each one taking a slot. When a slot is free it takes the next part off the channel and starts uploading it, and when an upload finishes its slot opens up again. It is done when the channel is closed and nothing is still running. *Figure: The upload engine, with the AbortGuard wrapped around all of it so that a failure anywhere cleans up after itself. Timings are illustrative.* Every multipart upload is wrapped in an `AbortGuard`, so if anything panics, errors or gets dropped halfway, the guard tells S3 to abort the upload on the way down. Without that, S3 keeps charging you for orphaned parts. ## Dropped parts and stale signatures The first real bug showed up as soon as the benchmark pushed real load: uploads would "complete" with chunks missing, and S3 would answer with `400 InvalidPart`. When an upload finished, the worker loop took its next job from the channel with `try_recv`, which doesn't wait. If the channel happened to be empty at that instant, the loop carried on with one fewer upload in flight, and once the in-flight set had drained, it **decided the work was over** and returned, with parts still on their way down the channel. The fix was to stop polling and use `tokio::select!` to wait on "a job arrived" and "an upload finished" at the same time, so the loop only ends when the channel is closed and nothing is left in flight. Simplified, the change looked like this: ```diff lang="rust" loop { - let Some(done) = tasks.join_next().await else { break }; - record(done); - if let Ok(job) = rx.try_recv() { - spawn(job); - } + tokio::select! { + Some(done) = tasks.join_next() => record(done), + Some(job) = rx.recv() => spawn(job), + else => break, + } } ``` *Figure: The same loop with each version, two uploads in flight and the scheduler a moment late with the rest.* While I was in there I found a second bug. A SigV4 signature includes a timestamp, and my retry loop was resending the *same signed request* on every attempt, so the timestamp got older with each backoff. S3 tolerates about **15 minutes of skew**, so it would have taken a big part on a slow link and a few retries to get there, but once a request is that old, S3 answers `RequestTimeTooSkewed`, and no number of further retries can fix that when each one carries the same stale signature. ## Pre-warming that slowed startup It's easy to assume that more concurrency means better performance, but that's not always true. I configured the client to open 256 connections immediately when it was created. The idea was to have all the connections ready before the first uploads started, so the uploads wouldn't have to wait for TLS handshakes. This seemed like a good optimization at first, so I made the constructor async and moved on. But opening all those connections upfront added **250 ms to 1 second** to every run. For a CLI that might be run hundreds of times, that startup cost adds up quickly. It also made the benchmark results inconsistent. Some runs finished the connection setup quickly, while others got stuck waiting for slower TLS handshakes. As a result, the benchmark had 2 groups of results instead of one consistent result. Then I realized that `reqwest` already has a connection pool that creates connections when they're needed. A new connection only took **around 2 ms**, which is tiny compared with the time spent waiting for an S3 request. So I removed my connection pre-warming. The result was pretty much exactly what I wanted: - Startup became faster. - The benchmark results became more consistent. - Throughput stayed the same. - The constructor could be synchronous again. ## 1.5 GiB of RAM for 40 MB files After dealing with slow startups, it was time for memory to teach me a lesson. Files under the multipart threshold (50 MiB at the time) took a simple path: read the whole file into memory and send it. Well, with 32 workers against a remote endpoint, where each upload sits in flight for hundreds of milliseconds, that added up to a peak of **1,566 MiB**, which is a lot of memory to spend on uploading some 40 MB files. I already knew the solution to this: lower the threshold and start streaming the file from disk in 256 KiB chunks, the same way the multipart path already did. This brought it down to **85 MiB** with no loss in throughput. ## Network shenanigans There are some bugs and optimizations that take a wizard to catch by reading the code, and I never went to Hogwarts. These ones I only found through my extensive, albeit ad hoc, testing, by running the real workload and watching what the kernel did with it. ### Slots that sat empty My first version of the `select!` fix from earlier waited for the next part *before* it checked for finished uploads, so a finished upload kept its slot until a new part showed up. Now it waits for both at once: ```rust tokio::select! { // an upload finished, its slot is free Some(done) = tasks.join_next() => record(done), // a part arrived and there is a free slot job = rx.recv(), if tasks.len() < slots => spawn(job), } ``` ### Everyone retrying together When S3 says "slow down", you wait and try again. All my workers waited the *same* amount, so they all came back together and got throttled again. A random wait, called jitter, spreads them out: ```diff lang="rust" -sleep(delay).await; +sleep(delay / 2 + random(0..=delay / 2)).await; ``` ### 40 ms on every write Every chunk I wrote to the socket stalled for about 40 ms. It was 2 TCP features waiting on each other, Nagle's algorithm on the sender and delayed ACKs on the receiver. Turning Nagle's algorithm off with `tcp_nodelay` sends the data right away. ### Asking which HTTP to speak Every new connection first agreed on an HTTP version with the server. S3 only speaks HTTP/1.1, so there was nothing to agree on. Forcing HTTP/1.1 skips the question. ## One writer thread per file Downloads had a problem of their own. A big download comes down as several parts at once, and each part writes its chunks straight into its own place in the file. That is what `pwrite` does, it writes at a position you give it. Writing to disk blocks, and async code shouldn't block, so `tokio` gives you [`spawn_blocking`](https://docs.rs/tokio/latest/tokio/task/fn.spawn_blocking.html), which runs the work on a separate pool of threads. My first version did that for every single write: ```rust // for every chunk of every part spawn_blocking(move || file.write_all_at(&chunk, offset)).await?; ``` That works, but for a 1 GB file it's thousands of trips through a thread pool that everything else in your program is sharing too. The OS also doesn't write to disk right away, it keeps the data in memory (the page cache) and flushes it later. When it decided to flush under all that write pressure, everything piled up behind it, and the same 5 GB download on the same network took anywhere from **4 to 29 seconds**. The fix was to give each file one thread that does nothing but write. The parts drop their chunks on a queue, and the thread writes them out one after another: ```rust std::thread::spawn(move || { while let Some(chunk) = queue.blocking_recv() { file.write_all_at(&chunk.buf, chunk.offset)?; } file.sync_all() }); ``` No more trip through the pool for each write, and nothing to fight over. The same 5 GB download now takes **4 to 6 seconds** *every* time. The queue has a fixed size, so if the disk falls behind the downloads wait, instead of chunks piling up in memory. *Figure: Four parts writing into one file. The dim cells are the rest of the program using the same pool. Timings are illustrative.* Two smaller wins came out of the same area: - **Fewer writes.** Each part now collects 512 KiB before sending it to the writer, which cut the writes from about 64,000 per GB to about 2,000. - **Fewer parts.** I had capped download parts at 256 MB, which S3 never asked for. Without the cap, a 10 GB file at concurrency 8 comes down as eight 1.25 GB streams, not forty 256 MB ones, so far fewer requests to sign and wait on. ## Choosing part size and concurrency This is the part which had a lot of back and forth. At first every part was a fixed 50 MiB, which is fine for some files and bad for most: | File | Parts at 50 MiB | Problem | | --- | --- | --- | | 50 MB | 1 | nothing to parallelise | | 10 GB | about 200 | a deep queue for 8 streams to chew through | | 600 GB | about 11,400 | over S3's limit of 10,000 parts | There is no defined formula for this, it's a problem of finding the right heuristic, and it was a lot of trial and error. For uploads I now aim for about **twice as many parts as streams**, because `PUT` latency varies and a bit of queue means a free stream always has the next part ready. For downloads I aim for one part per stream, because the server starts sending as soon as you ask and fewer `Range` requests win. The part size also grows with the file now, so the 10,000 limit is only in reach past **about 2.4 TiB**, and there s3z panics with a clear message, it doesn't start an upload it can't finish. Then I noticed the number of streams shouldn't be fixed either. A 128 MB file with 8 streams just allocates 8 buffers and barely uses them. Files up to 256 MiB now get 2 streams, up to 2 GiB get 4 and anything bigger gets 8. That dropped memory for the small-file workload from between 79 and 92 MB to **between 30 and 39 MB**, while big files still saturate the link. *Figure: How s3z splits a file today, from the same rules as the client's scheduler. Drag the size.* ## Python and Node bindings Creating Python and Node bindings was mainly an experiment to see if the library could work across multiple ecosystems while keeping one shared Rust core. The main challenge was that some Rust types borrowed data, which doesn't translate cleanly to Python or JavaScript. So I changed those types to own the data they needed, allowing the same API to work across Rust, Python and Node. Node also had a runtime issue, because [NAPI-RS](https://napi.rs) already manages a `tokio` runtime, so I had to reuse the existing runtime instead of creating another one. The result was one Rust implementation that could be used from Rust, Python and Node without maintaining separate implementations. ## You can make some backends happy some of the time I'd set up 4 S3-compatible backends locally ([MinIO](https://min.io), [RustFS](https://github.com/rustfs/rustfs), [SeaweedFS](https://github.com/seaweedfs/seaweedfs) and [Garage](https://garagehq.deuxfleurs.fr)) so I could catch my own protocol assumptions before users did, Hyrum's Law and all that. RustFS rejected every upload with `UnexpectedContent`, which took me far too long to trace to its wanting an explicit `Content-Length` header on every signed request, even where `reqwest` would have filled one in later anyway. I'd love to tell you all four stayed in the benchmark, but RustFS started hanging at random under load, and after half a day of trying to pin down why, I took it out rather than publish numbers from a backend that might hang in the middle of a run. ## A benchmark I can trust I did not expect to spend this much time on the benchmark, but the whole point of s3z is "this is faster", so the benchmark has to be something I trust. Every run gets a warmup, noisy runs get more samples until the numbers settle, and a compare command fails a change only if the difference is statistically real *and* bigger than the noise I've measured. Even so, it almost lied to me once. `mc` was losing badly against Garage and I nearly published a chart that said so, until I worked out that `mc` wasn't picking up the region from the environment the way the other tools were. The bug was in *my* setup. > **lesson** > > If a tool looks suspiciously bad in your benchmark, suspect your harness before > you suspect the tool. ## Results The reference run is committed in the repo: three 256 MB files, 10 to 30 runs per tool, against MinIO, SeaweedFS and Garage running in Docker on my laptop. On downloads s3z is level with `mc` and ahead of `s5cmd` and `aws-cli` on all 3 backends, and on uploads it's **ahead of all 3 tools everywhere**. Downloads come first, with MinIO shown and the tabs switching to the other backends: *Figure: Downloading 768 MB, three 256 MB files. Median of 10 to 30 runs per tool.* Uploads are where the transfer engine and all the fixes above pay off, with s3z ahead of `mc` by about a quarter of a second on MinIO and by well over a second on Garage: *Figure: Uploading the same 768 MB. Median of 10 to 30 runs per tool.* The part I'm proudest of, though, is how little memory it needs to get there: *Figure: Peak memory while downloading the same 768 MB from MinIO.* The 3 fixes from this post that moved the numbers most: - **85 MiB** upload memory, 40 MB files (from 1,566 MiB) −95% - **30–39 MB** small-upload memory (from 79–92 MB) −60% - **4–6 s** same 5 GB download (from 4–29 s) steady ## So much to break, so much to gain Tinkering with s3z humbled me. I realized how deep the ocean is, and my love for coral beads. I know they say cache invalidation and naming are the hardest things in computer science, but maybe they never had to deal with concurrency and contention. Either way, I am glad for all the pitfalls, and I am excited for all the things I'm yet to break. > All I know is that I know nothing. > > Socrates ## Try it The code is at [jaeaeich/s3z](https://github.com/jaeaeich/s3z). The CLI installs with one `curl`, the library is on crates.io, the Python wheels are on PyPI and the Node package is `@jae_aeich/s3z` on npm. There are also Axum, FastAPI and Elysia examples that all expose the same API, using s3z directly from Rust or through the bindings.