Poiesis v0.1: building a TES server to understand one

In the summer of 2024 I worked on TESK, the reference Kubernetes implementation of the GA4GH Task Execution Service, as a Google Summer of Code contributor with ELIXIR Cloud. I learned a lot from that codebase, and I also came out of it with a pile of opinions about what I would do differently if I started over.

The only way I could think of to test opinions like that was to start over, so I did. Poiesis is a TES server written in Python and built for Kubernetes from the first commit, and v0.1.0 went out earlier this month.

What TES is for

TES comes out of genomics, but the problem it solves is a plain platform one, and if you’ve ever had to glue a workflow engine to a cluster, you already know the pain this is about.

A research group runs an aligner, a variant caller, 3 QC steps and an annotator. Each step is a container, and a workflow engine (Nextflow, Snakemake, Cromwell, take your pick) wires them together. The whole thing then has to run on whatever the institution has: a Slurm queue, some HPC scheduler, a Kubernetes cluster in a cloud, sometimes all three depending on which grant paid for what.

The engine and the cluster don’t speak the same language, because the engine thinks in tasks while the cluster thinks in jobs, pods and Slurm scripts. Every engine therefore grew its own adaptor for every cluster manager, and the world ended up with N×M adaptors that nobody wants to maintain.

GA4GH (the Global Alliance for Genomics and Health) fixed this by putting one REST API in the middle: the engine submits tasks to a TES server, and the TES server runs them on whatever cluster it sits on. Write one TES server per kind of cluster, and any engine that speaks TES can use any of them.

1. 1engineNextflow, Cromwell
2. 2TES APIone contract
3. 3TES serverone per cluster
4. 4clusterKubernetes, Slurm

The contract is small and precise:

• A Task is an ordered list of Executors.
• An Executor means “run this image with this command and env, with these inputs on disk, and capture stdout and stderr.”
• Executors run strictly in order, so Executor 2 doesn’t start until Executor 1 has succeeded, and if Executor 1 fails the rest never run.
• Inputs are staged in from object storage (S3, HTTP) before the first Executor, and outputs are uploaded after the last.
• Every Task ends in one of 4 states: COMPLETE, EXECUTOR_ERROR, SYSTEM_ERROR or CANCELED.

The catch is that you’re running all of it on Kubernetes, which has its own ideas about containers and lifecycles that don’t quite line up with TES, and most of the work in a TES server goes into closing that gap.

Why not just use TESK

You can, and if you have a TES-shaped problem today and need something that runs, you probably should, because TESK works and runs in production at several institutions.

While I was inside the codebase, though, I kept noticing things I’d do differently:

• the data model carried history from earlier versions
• the Kubernetes side played it safe in places where newer primitives could have helped
• I wanted to design the operator side (Helm chart, secrets, RBAC) from a blank page instead of evolving it

I don’t mean any of that as a criticism of TESK, since it’s the same itch you get after months inside any codebase. You can scratch it by sending patches, or by building the thing you wish existed and bringing what you learn back, and I picked building because I wanted to see the design space without having to stay compatible with anything.

Poiesis is Greek for “the act of bringing something into being”, which felt right for a thing whose job is bringing tasks into being on a cluster. It was also free on PyPI, which with software names is rarer than it ought to be.

How v0.1 runs a task

A TES server on Kubernetes has to decide:

• where a task lives while it runs
• how its executors share a filesystem
• what enforces their order
• who watches them and writes their state down
• what happens when any of those pieces dies

v0.1’s answer to most of that is a small fleet of Kubernetes Jobs coordinated by a long-lived orchestrator, and its answer to the last question is down in the known problems.

When the API receives a CreateTask, it writes a Task record to MongoDB and starts a Pod called Torc (Task Orchestrator), which owns the task from then on and begins by creating a PVC (a Kubernetes volume, meaning a disk the task’s containers can share) sized to what the task asked for.

Torc then launches 3 workers, one after another:

• TIF (Task Input Filer). A Job that mounts the task’s PVC and stages every input onto it, pulling from S3, HTTP or the inline content field depending on the URL scheme.
• Texam (Task Executor and Monitor). A Job whose only purpose is to launch and watch your executors. It creates one Kubernetes Job per executor, in order, all mounting the same PVC, waits for each to finish before starting the next, and stops at the first non-zero exit.
• TOF (Task Output Filer). TIF in reverse: it walks the output list, expands globs and uploads each file to wherever its URL points. When it’s done, Torc writes the final state to Mongo and exits, and Kubernetes cleans up the rest.

1. 1TorcorchestratorPod
2. 2TIFPod
3. 3TexamPod
4. 4exec 1Pod
5. 5exec 2Pod
6. 6exec 3Pod
7. 7TOFPod

Redis pub/sub: each step reports back to Torc

7 steps: click one, or go through them with ›.1/7 · 1 Pod so far: the API starts Torc, which creates the task's volume and launches TIF.2/7 · 2 Pods so far: TIF stages the inputs onto the volume and publishes done on Redis.3/7 · 3 Pods so far: Torc hears TIF on Redis and launches Texam, which starts one Job per executor.4/7 · 4 Pods so far: Texam starts the first executor as its own Job and watches it.5/7 · 5 Pods so far: the second executor starts only once the first has succeeded.6/7 · 6 Pods so far: after the last executor, Texam reports back to Torc over Redis.7/7 · 7 Pods so far: Torc launches TOF to upload the outputs, waits for its done message, then writes the final state.

‹›↻

Torc and its children talk over Redis pub/sub: each child publishes a “done” message on a channel keyed by the task ID, and Torc blocks until it has heard from everyone it’s waiting on. I picked Redis because it’s cheap to run and does this “collect messages from N workers” pattern out of the box.

The names are deliberate, since Torc, TIF, Texam and TOF show up in logs, in Pod names and in metric labels. Once you know those 4 words, the whole system is easy to talk about, which helps a lot the first time somebody other than me has to debug it.

What I’d keep

Sequencing in Torc. I considered putting inputs, executors and outputs into one big Pod with N init containers, but something would still have to watch those containers and write their state down, and I couldn’t see where that watcher would live.

Making each phase its own Kubernetes object instead gives me clear boundaries and a failure story for each one (cleanup, restarts, knowing exactly which executor failed). Separate Jobs cost Pod startup time, which for tasks that usually run for minutes or hours is a trade I can live with, although it is also the first entry under known problems.

Pydantic end to end. Every request, every internal model and every stored object goes through Pydantic, with the TES schema, generated into code where possible, as the source of truth. The API rejects any task that doesn’t validate, which has already caught a good handful of my own bugs.

Pluggable storage. Inputs and outputs go through a FilerStrategy interface, with S3, HTTP and inline content in v0.1, so the first person who shows up wanting GCS has one obvious file to write. Outputs also accept globs, because real pipelines tend to produce “everything under /work/results/”, and nobody running one will put up with listing every file by hand for long.

Operator setup. The Helm chart is the supported install from day one, which made me design config, secrets, permissions and resource limits properly. Poiesis has no users or passwords of its own and trusts tokens from whatever OIDC login provider the institution already runs (there’s a dummy mode for local development that complains loudly in the logs so nobody ships it by accident).

Operators also set per-task defaults and limits for resources, node selectors and tolerations, and users can override the defaults within those limits, which sounds like a yak-shave right up until someone needs a GPU executor on a node that only accepts GPU work.

Known problems

Every task costs 4 + N Pods. That’s Torc, TIF, Texam and TOF, plus one per executor, so 5 executors means 9 Pods to schedule, and clusters with tight admission quotas will feel it. I don’t have a clean fix yet, because the obvious alternative, one Pod with init containers, has its own problem: something still has to watch the executors, and it’s not clear where that watcher would live.

Redis on the critical path. If Redis hiccups, tasks freeze, and operators have to run a Redis they understand and add it to their compliance scope. I’m accepting that because the alternatives (a custom controller, or heartbeats into the database) would have been a lot more code for a v0.1, but it’s a real cost, and sooner or later a serious operator is going to ask me to get rid of it.

MongoDB can’t enforce the TES schema. Pydantic checks it on the way in, but nothing checks it on the way out, so if the models drift, the stored documents drift with them and nothing says so. “Trust the application code” is a fine audit story for a v0.1 and a much worse one for a regulated environment.

Cancelling waits for the next Redis message. Cancel a task in the middle of a long executor and it stops when that executor finishes rather than when you pressed the button. There is a mitigation in the code, but it’s best effort, so sometimes it works and sometimes you wait.

No Kueue. Kueue, the Kubernetes project for queueing batch work fairly between teams, admits Jobs, and a Poiesis task is “the Torc Pod and everything it creates”, which isn’t one Job, so operators who want Kueue-style batch admission can’t have it in v0.1. This one bothers me the most, because it’s baked into the structure and no patch will fix it.

I’m putting this list on the public internet partly because it’s true, and partly so that the next time I open this codebase I can read it and decide what to fix first.

Where v0.1 stands

v0.1 follows the GA4GH TES 1.1 schema, has been tested as a Nextflow backend on a real cluster with a real pipeline, installs with one helm install and a Mongo connection string, and comes with end-to-end guides for a local cluster backed by MinIO and for generic OIDC.

It has run real tasks but not a million of them, and it isn’t built for tiny ones: 4 + N Pods is what it is, so 10,000 one-second tasks a minute will overwhelm it, whereas 100 tasks a minute that each run for minutes or hours will be fine. It also isn’t strictly multi-tenant, since there’s one database and one task namespace, so if you need hard isolation between tenants you should run one Poiesis per tenant.

The code is at jaeaeich/poiesis↗ under Apache 2.0, with the Helm chart in deployment/helm and the docs linked from the README, and the Nextflow guide is the quickest way to convince yourself it works. If you run TES on Kubernetes, or have opinions about how it ought to work, please open an issue, since I’ve spent enough months alone with this codebase that I’d like to hear where my taste is wrong.