Poiesis v0.2: from seven Pods per task to one

The first version of Poiesis ran every task as a small fleet. It had an orchestrator Pod, 3 worker Jobs, one more Job per executor, and Redis in the middle as a communication channel. This meant a task with 3 executors started 7 Pods and in v0.2 the same task runs as a single Pod - well sorta.

What a TES task is

Poiesis implements the GA4GH Task Execution Service (TES). If you haven’t met TES before, the v0.1 post has the long version, but here is the refresher:

• A Task is an ordered list of Executors, each a container with a command, env and some inputs on disk.
• Executors run strictly in order. If one fails, the rest never run.
• Inputs are staged in from S3 or HTTP before the first Executor, and outputs are uploaded after the last.
• Every Task ends as COMPLETE, EXECUTOR_ERROR, SYSTEM_ERROR or CANCELED.

v0.1: seven Pods per task

When I started, the obvious design was a small fleet where each component owned one part of a task’s life. Safe to say I took the single responsibility principle way too seriously. Anyone who has been near a batch system knows the pattern: an orchestrator, some workers, a message broker and a database.

The v0.1 fleet:

• API server. A stateless connexion app serving the TES REST API and writing Task records to MongoDB. As soon as a request came in, the API registered the task in the DB and returned 200.
• Torc (Task Orchestrator). One long-lived Pod per task, which creates the task’s shared volume (a PVC), launches the 3 workers in order and cleans up after them.
• TIF (Task Input Filer). A Job that stages inputs from S3 or HTTP onto the task’s PVC.
• Texam (Task Executor and Monitor). A Job that itself launches one Job per executor, in order, and watches each one using the Kubernetes watch API.
• TOF (Task Output Filer). A Job that, once every executor is done, uploads outputs from the PVC back to S3 or whatever other protocol is supported.
• MongoDB, holding tasks, executors and logs.
• Redis, as a message bus: each worker publishes “done” on a channel keyed by task ID, and Torc blocks until it hears the messages it expects.

Essentially, the API writes to Mongo and starts a Torc, which creates a PVC, launches TIF and blocks on Redis until TIF reports that it’s done. Torc then launches Texam, which starts the executor Jobs one at a time, watches each of them and reports back in turn, after which Torc launches TOF and waits once more. When TOF reports done, Torc writes the final state to Mongo and exits.

Where v0.1 hurt

1. Every task cost 4 + N Pods. A task with 3 executors started 7 (Torc, TIF, Texam, 3 executors and TOF), paying its own scheduling delay, image pull and container start. For a workflow engine sending hundreds of small tasks it adds up, and for the operator it meant the scheduler and admission quotas were busy with my orchestration instead of anyone’s actual work.

2. Redis was on the critical path. Every communication between the different components of a task happened through a Redis message, which made Redis the single point of failure. It also gave every operator one more stateful service to run and monitor.

3. When the orchestrator died, nobody noticed. Torc was the only thing that knew a task was in flight, so if it died after launching Texam but before hearing back, the children kept running, finished and published into the void, and the task sat in RUNNING in Mongo forever. There was no reconciler, and the “monitor” Job I wrote to paper over the worst cases didn’t change the fact that the hole was in the design.

There were smaller cuts too: MongoDB couldn’t enforce the TES schema, so the shape of stored documents slowly drifted; cancelling took effect “when the next pub/sub message arrived”, which in the middle of a long executor meant waiting for it to finish.

Init containers and native sidecars

I’d accepted 4 + N Pods per task; it was the natural design for running N containers in order, with inputs before and outputs after. The system seemed to need something to launch them and something to watch them, so I assumed that is what orchestration looked like. I couldn’t possibly reduce the linear growth of my Pods, because I needed another set of Pods/Jobs to watch over executions and report to the DB.

With an enormous amount of coffee and talking to LLMs, I discovered Kubernetes has native sidecars, which are init containers with restartPolicy: Always, and that they’ve been on by default since 1.29, which is, embarrassingly, a release older than the first commit of Poiesis.

A native sidecar starts before the init containers that follow it, stays up for the whole life of the Pod and is shut down after the main containers exit, which is exactly the shape of “something that watches the executors and writes down what happened”. Since any cluster a TES deployment realistically targets is past 1.29 by now, there wasn’t much left to design.

Kubernetes already has a primitive for “run these containers one at a time, stop at the first failure, and let them share a filesystem”, which is init containers, and they had been there all along.

Put it next to the TES list from earlier:

This lined up one for one, which means I had written a distributed orchestration layer to do, slowly and over Redis, what one Pod spec does on its own. This just makes me think how vast Kubernetes is and how beautiful the complexity and engineering behind it is.

v0.2: one Pod per task

In v0.2, one task is one Kubernetes Pod, wrapped in a Job. The Pod runs:

• TRec (Task Recorder). The native sidecar, which starts first and stays up for the life of the Pod, watches its own Pod through the Kubernetes API, and writes state changes and per-executor logs to Postgres as the other containers progress.
• TIF, as an init container, staging inputs onto the task’s PVC.
• N executor init containers, one per TES executor, kept in order by Kubernetes itself.
• TOF, as an init container, uploading outputs.
• ack, the main container. Kubernetes insists that a Pod has at least one regular container, even when all the real work happens in init containers, so the last thing in every task is a tiny step that reuses the Poiesis image, exits 0 and moves the Pod to Succeeded. It makes me happier than it should that every task ends with a container whose only job is to say “ok”.

kind: Job
spec:
  template:
    metadata:
      labels:
        poiesis.io/task: <task id>
    spec:
      restartPolicy: Never
      initContainers:
        - name: trec          # native sidecar, runs throughout
          restartPolicy: Always
        - name: tif           # only if there are inputs
        - name: exec-0
        - name: exec-1
        - name: exec-2
        - name: tof           # only if there are outputs
      containers:
        - name: ack           # exits 0: Pod Succeeded

All of them share one PVC, sized from the task’s disk_gb, and the PVC’s ownerReferences point at the Job, so that Kubernetes deletes the volume when it deletes the Job and we don’t have to clean it up manually.

Outside the TaskPod, one global piece is left. You would think the TaskPod is enough, as it does what a task needs to do (download data, process it and upload it), and the native sidecar is there to notice if a task ever fails. But there is another failure condition that isn’t covered, and this is where we introduce TCtl.

Normally the TRec writes the final state itself; TCtl exists for the cases where the TRec can’t, because the node died, the TRec itself got OOM-killed, or the Pod was evicted before it could finish. Argo Workflows and Tekton have the same shape, with an agent inside the Pod doing the work and a controller outside for when the agent can’t speak for itself.

• TCtl (Task Controller). A small Deployment with 2 or 3 replicas, only one of which is active at a time while the others wait to take over. It watches every Pod labelled poiesis.io/task, and if a TaskPod ends and Postgres has no final state for it, TCtl writes one, including the reason Kubernetes gave (OOMKilled, Evicted, Error).

The replicas agree on which one is active through a Kubernetes Lease: the leader keeps renewing it, and a standby only takes it over once it has gone stale.

That leaves one Pod per task, 3 components (the API, the TaskPod and TCtl) and 2 dependencies (Kubernetes and Postgres), which is the entire system.

What we deleted

Three components left the codebase in the redesign:

• Torc. Its only job was starting things in order and waiting on Redis, and init containers do both, with stronger guarantees.
• Texam. Its only job was starting one Job per executor and watching it, which is the same story; an executor is now just an entry in the TaskPod’s list of init containers.
• Redis. With nothing left to coordinate across Pods, Redis had nothing to do, so it’s gone from the chart and the docs, and operators no longer have to run it.

After the rewrite I ran vulture, a Python dead-code detector, over the repo, and it had a very good day.

From MongoDB to Postgres

The other half of v0.2 was moving off MongoDB, which ran and stored documents well enough but had no opinion about what those documents looked like. The TES schema, by contrast, is precise, with set fields on tasks, ordered executors and logs that nest in particular ways. In v0.1 nothing enforced any of that at the storage layer, since Pydantic checked it on the way in and nothing checked it after, so small drift piled up.

I ran v0.2 on Postgres with a proper relational schema:

• foreign keys
• NOT NULL constraints
• enum types for final states
• separate task_log, executor_log and system_log tables that match the spec one to one

Migrations are real migrations rather than “the code will probably write the new shape eventually”, and concurrent writes are safe because the database handles concurrency instead of a layer of hopeful application logic.

Smaller decisions

Read-only permissions. The TRec needs to watch its own Pod, so the TaskPod’s service account can get, list and watch Pods in the task namespace, with no writes and nothing cluster-wide, which means a user’s task can’t use those credentials to change anything.

No heartbeats. An earlier sketch had the TRec sending heartbeats and a sweeper killing tasks that went quiet, which was wrong, because a sweeper that only looks at the database can’t know why a task died. OOMKilled, Evicted and Error mean different things to the person who submitted the task, and only the Kubernetes API knows which one happened, so TCtl asks Kubernetes directly and writes the real reason to Postgres.

Cleanup through Kubernetes. ttlSecondsAfterFinished on the Job and ownerReferences from the PVC to the Job mean cleanup is configured once in the manifest and owned by Kubernetes, so there’s no reaper of mine to wonder about when a PVC seems to have outlived its task.

FastAPI replaced connexion. This was a boring rewrite with no performance story, since connexion’s spec-first approach was costing more effort than it saved and FastAPI’s Pydantic-native style matched how the rest of the code was already written, so it was mostly paying back an early framework choice.

What changed

Tasks start faster, because there’s 1 Pod to schedule instead of 7, and GET /tasks/{id} is accurate within seconds because the TRec writes as each container finishes. CANCELED now means cancelled, even in the middle of an executor, and SYSTEM_ERROR comes with the reason Kubernetes gave for the failure. On my side there’s much less code that I need to reason about.

The code is at jaeaeich/poiesis↗ under Apache 2.0, with the Helm chart in deployment/helm, and the cases I most want to hear about are the ones where one Pod per task doesn’t fit.