# Poiesis v0.2: from seven Pods per task to one

> Poiesis v0.1 ran a three-executor task as seven Pods held together by Redis, and v0.2 runs it as one, using a Kubernetes feature I’d talked myself out of.

Javed Habib · Mar 10, 2026 · #kubernetes #tes #ga4gh #python #architecture  
https://jaeaeich.com/writing/poiesis-rewrite

The first version of [Poiesis](https://github.com/jaeaeich/poiesis) ran every
task as a small fleet. It had an orchestrator Pod, 3 worker Jobs,
one more Job per executor, and Redis in the middle as a communication channel.
This meant a task with 3 executors started 7 Pods and in v0.2 the same
task runs as **a single Pod** - well sorta.

## What a TES task is

Poiesis implements the GA4GH **Task Execution Service** (TES). If you haven't
met TES before, the [v0.1 post](/writing/poiesis) has the long version, but here
is the refresher:

- A **Task** is an ordered list of **Executors**, each a container with a
  command, env and some inputs on disk.
- Executors run **strictly in order**. If one fails, the rest never run.
- **Inputs** are staged in from S3 or HTTP before the first Executor, and
  **outputs** are uploaded after the last.
- Every Task ends as `COMPLETE`, `EXECUTOR_ERROR`, `SYSTEM_ERROR` or
  `CANCELED`.

*Figure: A TES task: inputs, executors strictly in order, outputs, and the four ways it can end.*

## v0.1: seven Pods per task

When I started, the obvious design was a small fleet where each component owned
one part of a task's life. Safe to say I took the single responsibility
principle way too seriously. Anyone who has been near a batch system knows the
pattern: an orchestrator, some workers, a message broker and a database.

The v0.1 fleet:

- **API server.** A stateless
  [`connexion`](https://github.com/spec-first/connexion) app serving the TES
  REST API and writing Task records to MongoDB. As soon as a request came in,
  the API registered the task in the DB and returned `200`.
- **Torc** (Task Orchestrator). One long-lived Pod per task, which creates the
  task's shared volume (a PVC), launches the 3 workers in order and cleans
  up after them.
- **TIF** (Task Input Filer). A Job that stages inputs from S3 or HTTP onto the
  task's PVC.
- **Texam** (Task Executor and Monitor). A Job that itself launches
  **one Job per executor**, in order, and watches each one using the Kubernetes
  watch API.
- **TOF** (Task Output Filer). A Job that, once every executor is done, uploads
  outputs from the PVC back to S3 or whatever other protocol is supported.
- **MongoDB**, holding tasks, executors and logs.
- **Redis**, as a message bus: each worker publishes "done" on a channel keyed
  by task ID, and Torc blocks until it hears the messages it expects.

Essentially, the API writes to Mongo and starts a Torc, which creates a PVC,
launches TIF and blocks on Redis until TIF reports that it's done. Torc then
launches Texam, which starts the executor Jobs one at a time, watches each of
them and reports back in turn, after which Torc launches TOF and waits once
more. When TOF reports done, Torc writes the final state to Mongo and exits.

*Figure: v0.1: a task with three executors, which comes to seven Pods and one Redis.*

## Where v0.1 hurt

**1. Every task cost `4 + N` Pods.** A task with 3 executors started
**7** (Torc, TIF, Texam, 3 executors and TOF), paying its own scheduling
delay, image pull and container start. For a workflow engine sending hundreds of
small tasks it adds up, and for the operator it meant the scheduler and
admission quotas were busy with my orchestration instead of anyone's actual
work.

**2. Redis was on the critical path.** Every communication between the different
components of a task happened through a Redis message, which made Redis the
single point of failure. It also gave every operator one more stateful service
to run and monitor.

**3. When the orchestrator died, nobody noticed.** Torc was the only thing that
knew a task was in flight, so if it died after launching Texam but before
hearing back, the children kept running, finished and published into the void,
and the task sat in `RUNNING` in Mongo forever. There was no reconciler, and the
"monitor" Job I wrote to paper over the worst cases didn't change the fact that
the hole was in the design.

There were smaller cuts too: MongoDB couldn't enforce the TES schema, so the
shape of stored documents slowly drifted; cancelling took effect "when the next
pub/sub message arrived", which in the middle of a long executor meant waiting
for it to finish.

## Init containers and native sidecars

I'd accepted `4 + N` Pods per task; it was the natural design for running N
containers in order, with inputs before and outputs after. The system seemed to
need something to launch them and something to watch them, so I assumed that is
what orchestration looked like. I couldn't possibly reduce the linear growth of
my Pods, because I needed another set of Pods/Jobs to watch over executions and
report to the DB.

With an enormous amount of coffee and talking to LLMs, I discovered Kubernetes
has
[native sidecars](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/),
which are init containers with `restartPolicy: Always`, and that they've been
**on by default since 1.29**, which is, embarrassingly, a release older than the
first commit of Poiesis.

A native sidecar starts before the init containers that follow it, stays up for
the whole life of the Pod and is shut down after the main containers exit, which
is exactly the shape of "something that watches the executors and writes down
what happened". Since any cluster a TES deployment realistically targets is past
1.29 by now, there wasn't much left to design.

Kubernetes already has a primitive for "run these containers one at a time, stop
at the first failure, and let them share a filesystem", which is
[init containers](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/),
and **they had been there all along**.

Put it next to the TES list from earlier:

| TES asks for | Init containers already do |
| --- | --- |
| executors run in order | run in the order you declare them |
| stop at the first failed executor | stop at the first failure |
| executors share a working directory | share the Pod's volumes |
| inputs before executors, outputs after | put input and output steps first and last in the list |

This lined up one for one, which means I had written a distributed orchestration
layer to do, slowly and over Redis, what **one Pod spec does on its own**. This
just makes me think how vast Kubernetes is and how beautiful the complexity and
engineering behind it is.

## v0.2: one Pod per task

In v0.2, **one task is one Kubernetes Pod**, wrapped in a Job. The Pod runs:

- **TRec** (Task Recorder). The native sidecar, which starts first and
  stays up for the life of the Pod, watches its own Pod through the Kubernetes
  API, and writes state changes and per-executor logs to Postgres as the other
  containers progress.
- **TIF**, as an init container, staging inputs onto the task's PVC.
- **N executor init containers**, one per TES executor, kept in order
  by Kubernetes itself.
- **TOF**, as an init container, uploading outputs.
- **`ack`**, the main container. Kubernetes insists that a Pod has at least one
  regular container, even when all the real work happens in init containers, so
  the last thing in every task is a tiny step that reuses the Poiesis image,
  exits `0` and moves the Pod to `Succeeded`. It makes me happier than it should
  that every task ends with a container whose only job is to say "ok".

*Figure: v0.2: the same task as one Pod, with the recorder running alongside every step.*

```yaml title="TaskPod, simplified"
kind: Job
spec:
  template:
    metadata:
      labels:
        poiesis.io/task: <task id>
    spec:
      restartPolicy: Never
      initContainers:
        - name: trec          # native sidecar, runs throughout
          restartPolicy: Always
        - name: tif           # only if there are inputs
        - name: exec-0
        - name: exec-1
        - name: exec-2
        - name: tof           # only if there are outputs
      containers:
        - name: ack           # exits 0: Pod Succeeded
```

All of them share one PVC, sized from the task's `disk_gb`, and the PVC's
`ownerReferences` point at the Job, so that Kubernetes deletes the volume when
it deletes the Job and we don't have to clean it up manually.

Outside the TaskPod, one global piece is left. You would think the TaskPod is
enough, as it does what a task needs to do (download data, process it and upload
it), and the native sidecar is there to notice if a task ever fails. But there
is **another failure condition that isn't covered**, and this is where we
introduce `TCtl`.

Normally the TRec writes the final state itself; TCtl exists for the cases where
the TRec can't, because the node died, the TRec itself got OOM-killed, or the
Pod was evicted before it could finish. [Argo
Workflows](https://argoproj.github.io/workflows/) and
[Tekton](https://tekton.dev) have the same shape, with an agent inside the Pod
doing the work and a controller outside for when the agent can't speak for
itself.

- **TCtl** (Task Controller). A small Deployment with 2 or 3 replicas, only one
  of which is active at a time while the others wait to take over. It watches
  every Pod labelled `poiesis.io/task`, and if a TaskPod ends and Postgres has
  no final state for it, TCtl writes one, including the reason Kubernetes gave
  (`OOMKilled`, `Evicted`, `Error`).

The replicas agree on which one is active through a Kubernetes
[Lease](https://kubernetes.io/docs/concepts/architecture/leases/): the leader
keeps renewing it, and a standby only takes it over once it has gone stale.

*Figure: TCtl's leader election, with time running four times as fast: the leader renews the Lease every 5 seconds, and when it dies a standby takes over 30 seconds later.*

That leaves one Pod per task, 3 components (the API, the TaskPod and TCtl) and
2 dependencies (Kubernetes and Postgres), which is the entire system.

## What we deleted

Three components left the codebase in the redesign:

- **Torc.** Its only job was starting things in order and waiting on Redis, and
  init containers do both, with stronger guarantees.
- **Texam.** Its only job was starting one Job per executor and watching it,
  which is the same story; an executor is now just an entry in the TaskPod's
  list of init containers.
- **Redis.** With nothing left to coordinate across Pods, Redis had nothing to
  do, so it's gone from the chart and the docs, and operators no longer have to
  run it.

After the rewrite I ran [vulture](https://github.com/jendrikseipp/vulture), a
Python dead-code detector, over the repo, and it had a very good day.

## From MongoDB to Postgres

The other half of v0.2 was moving off MongoDB, which ran and stored documents
well enough but had no opinion about what those documents looked like. The TES
schema, by contrast, is precise, with set fields on tasks, ordered executors and
logs that nest in particular ways. In v0.1 nothing enforced any of that at the
storage layer, since Pydantic checked it on the way in and nothing checked it
after, so small drift piled up.

I ran v0.2 on Postgres with a proper relational schema:

- foreign keys
- `NOT NULL` constraints
- enum types for final states
- separate `task_log`, `executor_log` and `system_log` tables that match the
  spec one to one

Migrations are real migrations rather than "the code will probably write the new
shape eventually", and concurrent writes are safe because the database handles
concurrency instead of a layer of hopeful application logic.

## Smaller decisions

**Read-only permissions.** The TRec needs to watch its own Pod, so the TaskPod's
service account can get, list and watch Pods in the task namespace, with no
writes and nothing cluster-wide, which means a user's task can't use those
credentials to change anything.

**No heartbeats.** An earlier sketch had the TRec sending heartbeats and a
sweeper killing tasks that went quiet, which was wrong, because a sweeper that
only looks at the database can't know *why* a task died. `OOMKilled`, `Evicted`
and `Error` mean different things to the person who submitted the task, and only
the Kubernetes API knows which one happened, so TCtl asks Kubernetes directly
and writes the real reason to Postgres.

**Cleanup through Kubernetes.** `ttlSecondsAfterFinished` on the Job and
`ownerReferences` from the PVC to the Job mean cleanup is configured once in the
manifest and owned by Kubernetes, so there's no reaper of mine to wonder about
when a PVC seems to have outlived its task.

**[FastAPI](https://fastapi.tiangolo.com) replaced `connexion`.** This was a
boring rewrite with no performance story, since `connexion`'s spec-first
approach was costing more effort than it saved and FastAPI's Pydantic-native
style matched how the rest of the code was already written, so it was mostly
paying back an early framework choice.

## What changed

- **1** Pods for a three-executor task (from 7)
- **Postgres** stateful dependencies (from MongoDB + Redis)

Tasks start faster, because there's 1 Pod to schedule instead of 7, and
`GET /tasks/{id}` is accurate within seconds because the TRec writes as each
container finishes. `CANCELED` now means *cancelled*, even in the middle of an
executor, and `SYSTEM_ERROR` comes with the reason Kubernetes gave for the
failure. On my side there's much less code that I need to reason about.

The code is at [jaeaeich/poiesis](https://github.com/jaeaeich/poiesis) under Apache 2.0, with the Helm
chart in `deployment/helm`, and the cases I most want to hear about are the ones
where one Pod per task doesn't fit.