Case study
Kiln-IDP
An internal developer platform: a request goes in, a hardened resource comes out, and a cost-aware scheduler keeps latency-critical work off capacity that can vanish.
Results
- latency-class tasks on spot
- 057,012 replayed placements (521 latency-class) on 12,477 machines from the public Google 2011 cluster trace, at every spot fraction. The default scheduling model put 138–384 there.
- p50 self-service provisioning
- 11s16s p95, n=10, on a kind cluster in CI.
- worst false-rollback rate
- 2.8%1,000 simulated canary rollouts per traffic class.
- 1,000 rollouts promoted
- 3m58sPreviously hit a 12-minute timeout, until the controller-runtime 20 qps / 30 burst client limit was lifted.
Problem
Without a platform layer, a team gets infrastructure one of two ways. They file a ticket with a platform engineer, which takes hours to days and waits on a person being free. Or they write the Terraform and kubectl themselves, which is fast and inherits every way there is to get security, cost and reliability wrong.
Neither path leaves a real audit trail for when compliance asks who provisioned what. And neither knows that a latency-sensitive service should never land on spot capacity, which is cheap precisely because the provider can take it back at any moment.
What I built
- 01APISpring Boot · RS256 JWT, role check
- 02Policy gateKyverno at admission, fail-closed
- 03ProvisionCrossplane v2 + ArgoCD + operator
- 04Schedulekiln-scheduler CostAware plugin
- 05DeliverIstio canary, sequential-test rollback
- 06AuditKafka → Postgres hash chain
Every request is authenticated, checked against policy before anything is composed, provisioned through GitOps, and traced end to end as a single OpenTelemetry trace. Every subsystem publishes to one Kafka audit topic, and only the audit service can write the table, which is a hash chain, so a missing or edited row shows.
The scheduler is a kube-scheduler plugin. A pod's workload class comes from a label: latency-sensitive, standard or batch. The guarantee is a Filter, not a score: a latency-sensitive pod sees spot nodes as unschedulable, so no weighting can trade it away. Only then does scoring run: 50% cost, 30% packing (with a penalty for stranding GPUs), 20% preemption risk.
Both defaults fail safe. A pod with no class is treated as latency-sensitive. A node with no pricing is treated as on-demand at the highest known cost.
Around it: a canary controller that decides with a Wald sequential probability ratio test and CUSUM change detection instead of a fixed per-window threshold, and a chaos controller whose faults are held by a lease, so a crashed controller cannot leave a fault running.
Considered and set aside
- Karpenter—decides which nodes exist, not which node a pod lands on
- Affinity and taints—can only filter; there is no cost term to optimize
- Argo Rollouts / Flagger—judge each window against a threshold rather than accumulating evidence
- Chaos Mesh / Litmus—no lease, so a fault outlives a crashed controller
Reading the numbers
The cost result is the one to read carefully. On an empty cluster kiln billed 91.8% less, but the trace's cluster is already 92% full, and there the saving is 0%. The empty-cluster figure is an upper bound, not a result. The guarantee is the headline, not the savings.
The 57,012-placement replay exercises the scoring model. The real kube-scheduler binary with the plugin was run on the same trace against an empty cluster only (0 latency-class on spot, against 257 for the default); the full-cluster binary run did not finish.
The trace has no prices. Half the machines were labelled spot at synthetic prices, and priority 9 and above was mapped to latency-sensitive.
Lessons
- Make the safety property a filter, not a weight. Anything that can be scored can be outscored.
- Report the number that holds under realistic conditions. The 91.8% looked like the headline until the replay ran against the cluster as it really was.
- Write down where the system loses. The canary controller's own report lists 7.5% false rollbacks on retry storms and 27.9% missed regressions on heavy-tailed latency.
NextAn Ed25519-signed external anchor for the audit hash chain, and running the GKE configuration that is written but has never been applied.