Production Template¶
Production template and agentic operating model
A reusable ML service template with governed AI-assisted engineering¶
The ML Service Template is the strongest artifact in this portfolio. It packages the production lessons from the monorepo into a starter system for ML services: FastAPI serving, training/serving parity, CI/CD, Docker, Kubernetes, Terraform examples, observability hooks, runbooks and an explicit agentic governance model. The template also encodes 38 anti-patterns with corrective actions, SLSA L2 supply-chain security practices, closed-loop monitoring with statistical promotion gates, native-cloud edge protection (Cloud Armor / AWS WAF+Shield, Cloudflare optional), and a self-auditing documentation-coherence system that keeps its own version, ADR count and governance surface honest across every document — a gate, not a suggestion.
Its scope limit is a decision, not an omission
This template governs classical ML — scikit-learn, XGBoost, LightGBM — for a single team running 1–5 models, with small-team calibration throughout — "2–3 models → CronJob, not Airflow"; "in-memory DataFrames → Pandera, not Great Expectations". Widening it to cover feature stores, lakehouse table formats, distributed training, GenAI serving and multi-project orchestration would not improve it: it would destroy the property that makes it recommendable, which is that it is small enough to read in an afternoon.
That work sits above this boundary, and it got its own repository —
ml-platform, the second and enterprise template.
It consumes this one through copier rather than
replacing it: where the two disagree about serving, containers, probes,
manifests or supply chain, this template wins.
See the two side by side →
How To Read This Template¶
Recruiter view
Reusable system, not only a project¶
The important signal is that portfolio lessons became a repeatable starter system for future ML services, with defaults, guardrails and documentation.
Technical lead view
Inspect the operating contracts¶
The best evidence is in the rules, skills, workflows, manifest and anti-pattern catalog that constrain how services are generated and operated.
Team adoption view
Can another engineer use it?¶
The template is designed around quick start, service scaffold, tests, deployment artifacts and reviewable AI-assisted workflows.
Code Review Shortcuts¶
Service Dockerfile K8s deployment CI template Deploy GCP Agent manifest Agent rules Agent skills Agent workflows
Why This Is More Than A Template¶
The template is not only a folder structure. It is an operating contract for starting ML services with fewer avoidable mistakes. It defines what a generated service should contain, how it should be validated, which deployment paths it can follow and how AI agents are allowed to participate in the work.
The important point is the second-order artifact: I did not only build three portfolio services. I extracted the reusable production system behind them.
Technical reviewer signal
The most distinctive part is the agentic operating model: canonical rules, skills, workflows and risk escalation are written as code-adjacent governance, not left as informal prompting.
Production Scaffold Contract¶
1. Service API
FastAPI by default¶
The scaffold includes prediction endpoints, batch prediction, readiness, health, metrics, error envelopes, auth hooks and model metadata.
2. ML lifecycle
Training to serving parity¶
Feature engineering, schema validation, model loading and inference contracts are treated as one path instead of separate notebook and API worlds.
3. Quality gates
CI before adoption¶
YAML checks, workflow checks, scaffold tests, smoke paths, pre-commit hooks and targeted validations protect the generated service.
4. Runtime
Docker and Kubernetes¶
Containers, Kustomize overlays, readiness probes and deployment defaults are part of the service contract from the beginning.
5. Observability
Metrics and operations hooks¶
Prometheus-compatible metrics, prediction logging, tracing options, drift checks and retraining workflows are built into the template story. A six-station coverage audit (edge, infra, inference, models, logs/traces, business KPIs) closed the real gaps it found: executor-saturation metrics, a business-KPI dashboard, and a structured access log that actually correlates a request to its trace on every call, not only on failures.
6. Cloud path
GCP and AWS patterns¶
The template documents GKE/EKS deployment expectations, identity patterns, artifact registries and operational runbooks without pretending local tests are cloud proof. An opt-in edge layer (Cloud Armor / AWS WAF+Shield, Cloudflare optional) sits in front of the Ingress once an adopter wires it in.
How It Works, In Four Diagrams¶
The contracts above are written out in prose in the repository, because a
contract needs prose. The first hour does not. These four diagrams are the
shortest honest description of what the template actually does — the deploy
chain, identity and secrets, the monitoring loop, and what governs the agentic
surface. Each one describes behaviour that exists in the tree today; the same
four, with the file that implements every step, live in
docs/DIAGRAMS.md.
1. From a commit to a pod you can verify¶
One property matters before any other: the thing that is signed, the thing that is scanned and the thing the cluster admits are the same bytes, because every step after the build addresses the image by digest rather than by tag. A tag can be moved; a digest cannot.
flowchart TD
A["git push, or a vX.Y.Z tag"] --> B["CI: lint, tests, contract check, security audit"]
B --> C["build job authenticates by OIDC<br/>no static cloud keys anywhere"]
C --> D["docker build and push"]
D --> E["resolve the manifest digest"]
E --> F["cosign sign, by digest"]
E --> G["syft generates a CycloneDX SBOM"]
G --> H["cosign attest the SBOM, by digest"]
F --> I["one reusable deploy job per environment"]
H --> I
I --> J["Environment protection<br/>dev auto, staging and prod gated, prod tag-only"]
J --> K["live-signal pre-check<br/>risk can escalate the mode to STOP and abort"]
K --> L["the overlay image is pinned to that digest"]
L --> M["apply to the target overlay"]
M --> N["Kyverno admission: digest required, signature verified"]
N --> O["rollout status, 600s budget"]
O --> P["smoke test: readiness, then the auth path<br/>with a deliberately wrong key"]
P --> Q["audit entry appended"]
Two nodes are the adopter's to wire, and the diagram would be dishonest without saying so: reviewer counts live in GitHub's Environment settings rather than in the repository, and Kyverno admits nothing until its policies are installed in the cluster. The tag-only gate on production is enforced in the repository, twice.
Two other steps exist because the obvious version failed. Building and
signing a tag leaves a window in which the tag can move, so the build job
emits a digest map and the deploy job rewrites the overlay with it. And a
readiness probe alone once went green on a deployment whose every
authenticated request failed — readiness never touches the secret backend — so
the smoke test sends a key that is deliberately wrong: 401 proves the secret
resolved, 503 proves it did not.
2. Who the code is, and where its secrets come from¶
There is no credential file anywhere in this template: not in the repository, not in the image, not on the node. Every actor proves its identity with a short-lived token, and the identities are deliberately separate — the job that detects drift cannot read the API key, and the job that retrains cannot delete a model.
flowchart LR
subgraph TF["Terraform, per environment"]
T1["federation pool and provider<br/>trust conditioned on this repository"]
T2["five purpose-scoped identities<br/>ci, deploy, runtime, drift, retrain"]
T3["secret entries<br/>only the runtime identity is granted access"]
end
subgraph GH["GitHub Actions"]
G1["OIDC token<br/>subject names the repo and the environment"]
G2["exchanged for a cloud identity<br/>Workload Identity on GCP, IRSA on AWS"]
end
subgraph K8S["Cluster"]
K1["ServiceAccount annotated with that identity"]
K2["pod starts with no key on disk"]
K3["the secret is resolved at runtime and cached"]
end
T1 --> G1
G1 --> G2
T2 --> G2
T2 --> K1
K1 --> K2
K2 --> K3
T3 --> K3
K3 --> R["request served, or 503 if the backend is unreachable"]
The part that is easy to get wrong is the name. Terraform decides what a
secret is called and the pod has to ask for the same string — joined with -
on GCP and / on AWS. That join lives in one function, and a contract test
evaluates the Terraform interpolations against the Kubernetes overlays so four
layers cannot quietly choose four different names. Failure is closed: if the
backend is unreachable in staging or production the service answers 503
rather than falling back to an unauthenticated path.
3. The loop that closes¶
A monitoring setup that only emits metrics is an open loop — it can say something changed, but nothing downstream is obliged to act. This one closes, and it closes through artefacts a human reads: an issue, a quality-gate report, a champion-versus-challenger decision. Not an automatic push to production.
flowchart TD
R["POST /predict"] --> S["inference on a thread pool<br/>the event loop is never blocked"]
S --> T["the prediction is logged to object storage"]
S --> U["Prometheus metrics, SLO rules, AlertManager"]
T --> V["ground-truth ingester, daily"]
V --> W["performance monitor, daily<br/>sliced metrics pushed to the gateway"]
T --> X["drift job, daily<br/>PSI over quantile bins"]
X -->|"drift above threshold"| Y["an issue is opened"]
W -->|"sliced performance degrades"| Y
Y --> Z["retrain workflow, dispatched by a human"]
Z --> AA["schema validation, training, quality gates<br/>primary metric, fairness ratio, leakage check"]
AA --> AB["champion versus challenger, statistically compared"]
AB -->|"promote"| AC["model signed and registered<br/>re-enters diagram 1"]
AB -->|"keep or blocked"| AD["promotion refused, decision recorded"]
The fairness gate uses a disparate impact ratio with a floor of 0.80, the US
four-fifths rule — and the template says in the same file that this is a
starting point rather than a universal threshold, that the metric is
group-level, that it is unreliable on small subgroups, and that passing it
does not by itself make a model fair.
4. What governs the agent¶
The agentic surface is not load-bearing. Agents make the work faster; the gates decide what ships. Everything an agent reads is generated from one canonical tree, and a check fails CI when a generated adapter stops matching its source — so a rule cannot be softened in one IDE's copy without the difference appearing in a diff.
flowchart TD
A["AGENTS.md<br/>canonical policy and the anti-pattern catalogue"] --> B["the agentic manifest"]
B --> C["rules"]
B --> D["skills"]
B --> E["workflows"]
C --> F["one generated adapter per supported tool<br/>plus a byte-identical mirror in the payload"]
D --> F
E --> F
F --> G["adapter sync check<br/>a drifted adapter fails CI"]
A --> H["deterministic gates"]
H --> I["doc coherence, traceable control claims, pin shape,<br/>deploy contract, payload scope, render safety, and more"]
I --> J["CI lanes: validate templates, policy tests, context tests"]
G --> J
J --> K["what ships is what the gates allowed"]
The diagram names no IDE on purpose. Its first version listed two, and within a week the template began rendering skills where Cursor and Codex actually look for them — the picture was wrong before anyone had edited it. Which tools are supported is an inventory that changes whenever a tool does, so it lives in the template's AGENTS.md, where a gate reconciles every count against the directories. A diagram draws the mechanism, which changes rarely.
The same idea applies to the page you are reading. The template's coherence gate reconciles the claims that appear in more than one document — the version, the anti-pattern count, the agentic surface counts, ADR numbering — against the tree, in every live document at once rather than in a list of filenames someone has to remember to extend.
Agentic Operating Model¶
Canonical rules
One source of truth¶
AGENTS.md, agent context and the manifest define the behavior contract.
Adapter files point to canonical rules instead of drifting into parallel policy.
Risk matrix
AUTO / CONSULT / STOP¶
Low-risk work can be automated, ambiguous production work requires user consultation, and destructive or safety-sensitive actions must stop.
Workflow skills
Repeatable MLOps actions¶
New service creation, drift checks, retraining, rollback, cost review, release, incident and security workflows are encoded as reusable agent skills.
Reviewability
Agents leave evidence¶
The model is designed around validation logs, changelogs, ADRs, runbooks and explicit test commands so agent-assisted work remains auditable.
Rules, Skills And Workflows¶
Rules (19)
Context-aware engineering constraints¶
Rules cover Python serving, training, Kubernetes, Terraform, Docker, GitHub Actions, monitoring, data validation, security, API contracts, Copier template lifecycle, documentation coherence and edge protection (Cloud Armor / AWS WAF+Shield / optional Cloudflare). The goal is to make failure modes harder to reintroduce.
Skills (27)
Reusable MLOps procedures¶
Skills include new service creation, EDA, deploy to GKE/EKS, drift checks, model retraining, release checklist, rollback, cost audit, security audit, incident response, stack-profile switching, adopter onboarding, documentation-coherence enforcement, CI-green verification, dual-axis PR review, systematic bug diagnosis, pre-scaffold ML problem spec capture, blameless incident postmortems and edge-protection coverage auditing.
Workflows (20)
Slash-command operating paths¶
Workflows such as /new-service, /incident,
/release, /drift-check, /retrain,
/rollback, /secret-breach, /stack-switch,
/onboard, /doc-coherence, /ci-green and
/edge-setup turn repeatable MLOps work into auditable steps.
Anti-Pattern Catalog¶
Serving
No uvicorn --workers N under Kubernetes¶
Corrective action: one worker per pod, HPA for horizontal
scaling, and ThreadPoolExecutor for CPU-bound inference.
Autoscaling
No memory-based HPA for ML pods¶
Corrective action: use CPU as the scaling signal because loaded models keep a fixed memory footprint even when traffic drops.
Async APIs
No direct model.predict() in async endpoints¶
Corrective action: move CPU-bound prediction work behind
asyncio.run_in_executor() so request handling stays responsive.
Explainability
No TreeExplainer for StackingClassifier¶
Corrective action: use KernelExplainer with a predict-proba wrapper in the original feature space.
Packaging
No model artifacts baked into Docker images¶
Corrective action: keep images immutable and load model artifacts through runtime storage patterns such as init containers.
Security
No static cloud credentials in production paths¶
Corrective action: use Workload Identity on GCP and IRSA on AWS, with CI/deploy/runtime identities separated by purpose.
Adoption safety
No cloud credentials in a "local" stack profile¶
Corrective action: a local profile must
structurally refuse cloud credentials, Kubernetes and Docker — enforced by
a contract test and a runtime guard in make deploy, not just
a naming convention.
Release safety
No promoting or deploying on red/missing CI¶
Corrective action: a read-only skill verifies CI status before release or a staging/prod deploy; overriding a red or missing signal requires explicit human approval and an audit-trail entry (D-36).
Edge exposure
No public Ingress without edge protection¶
Corrective action: a production overlay must wire in Cloud Armor or AWS WAF+Shield (Cloudflare optional) before going live; disabling an existing WAF/rate-limit rule is a STOP-class action in every environment, no exceptions (D-38).
Adoption Engineering & Self-Auditing Documentation¶
Governance only counts if adopting the template is actually easy. A later
pass (v0.19.0–v0.20.0)
rebuilt the scaffolding path on Copier
(so copier update can pull template improvements into an
already-adopted service), added local-first stack profiles so a reviewer
can evaluate the whole train → serve → drift loop without provisioning a
cluster, and mapped the production layout to the Cookiecutter Data
Science vocabulary for practitioners coming from a notebook-first
background.
The differentiated piece is the last one: a documentation coherence system that treats "the docs agree with reality" as a CI-enforced contract, not a hope.
Technical reviewer signal
An independent audit of this exact work found and fixed a real bug before release: an onboarding flow was validating its output against the wrong JSON schema and would have failed on first use. The fix, and four other real defects, are documented in the release notes instead of quietly folded in — that disclosure habit is the same one the template asks of its adopters.
Scaffolding
Copier, not cp + sed¶
A custom Jinja delimiter ({@ @}) avoids collisions with the
literal ${{ }} GitHub Actions syntax the template ships. Two
anti-patterns (D-33, D-34) keep the scaffolder from regressing to manual
substitution.
Local-first profiles
local / staging / prod¶
Chosen at scaffold time; local runs the full loop with zero
cloud dependencies. Switching profiles is a reviewable, CONSULT-mode
operation, never a silent edit.
Recognizable layout
A generated CCDS mapping¶
A documentation-only view translates the production directory layout into Cookiecutter Data Science vocabulary — no directories renamed, no production path touched.
Self-auditing docs
A CI gate for the docs themselves¶
One deterministic script enforces a single source of truth for release version, anti-pattern count, agentic-surface counts and ADR numbering across every document — a gate, not a suggestion.
Enterprise Governance & Compliance Posture¶
A later benchmarking pass compared the template against NIST AI RMF, ISO/IEC 42001, the EU AI Act and frontier open-source scaffolds — not against the author's other repos. The finding: the template already produces the evidence these frameworks ask for (quality gates, fairness thresholds, an audit trail, human-in-the-loop approval). What it lacked was the map connecting that evidence to each framework's own vocabulary.
The discipline that matters most
The compliance document is explicit about what it is not: not a certification, not a substitute for legal review, not a claim that any template can be "AI Act compliant" — only a deployed, operated system can be evaluated. Declining to over-claim is itself the signal.
Compliance mapping
NIST AI RMF · ISO/IEC 42001 · EU AI Act¶
Traces artifacts the template already produces — quality gates, the fairness DIR floor, the audit trail, AUTO/CONSULT/STOP human oversight — to each framework's own control questions. Descriptive, never certifying.
Supply chain
SHA-pinned CI + OpenSSF Scorecard¶
Every GitHub Action across every workflow is pinned by commit SHA, not a mutable tag. A dedicated Scorecard workflow scores the repo against OpenSSF's supply-chain criteria on every push.
Agentic governance pattern
Read is AUTO, override is STOP¶
The newest gate (D-36) separates "check whether CI is green" (always allowed, read-only) from "proceed despite red or missing CI" (an explicit human STOP-class approval, logged to the audit trail) — wired as a hard precondition into release and staging/prod deploy.
Portability
Escape hatches, not lock-in¶
A documented swap matrix for cloud, experiment tracking, serving backend, IaC engine and scaffolding tool — so "agnostic to technologies" is a verifiable claim, not a slogan.
Edge protection
Native-cloud-first, Cloudflare optional¶
Cloud Armor and AWS WAF+Shield Standard are the default per-cloud WAF and DDoS layer; Cloudflare stays available for genuinely concurrent multi-cloud deployments, but is never the default — the common case is one cloud, not a third-party account layered on top of it.
How It Compares¶
None of the well-known alternatives occupy quite the same spot. Each is genuinely strong at what it optimizes for — the comparison below is about fit, not a claim that this template is universally "better."
The honest version
This is a hardening baseline for a small-to-mid ML team that wants production defaults without adopting a control-plane. It is not a managed pipeline product, not a full MLOps platform, and not a drop-in replacement for an already-mature internal platform team.
| Alternative | Strong at | What this template adds |
|---|---|---|
| Cookiecutter Data Science | The most recognized project-layout convention; huge familiarity | CCDS gives you a folder shape. It has no serving contract, no CI/CD, no Kubernetes/Terraform, no supply-chain security and no agentic governance — this template maps to CCDS vocabulary for practitioners coming from it (see Adoption Engineering above) rather than competing with it. |
| Kubeflow | Full ML platform: pipelines, serving, multi-tenancy, a real control plane | Kubeflow needs a dedicated operator and real operational investment before the first service ships. This template is a starter a small team can scaffold from cold in minutes, with the option to grow into a platform like Kubeflow later — not a prerequisite for one. |
| MLRun / ZenML / Metaflow | Strong pipeline/DAG orchestration abstractions, good experiment ergonomics | These orchestrate the ML workflow; none of them ship Kubernetes manifests, Terraform, signed-image CI/CD, or an agentic governance layer as part of the same starter — you still assemble the production surface yourself. |
| Cloud-native pipelines (Vertex AI Pipelines, SageMaker Pipelines) | Managed, low operational burden, deep integration with one cloud | Powerful, but cloud-locked — you build on the platform, not with portable source you own. This template ships working Terraform + Kustomize for both GCP and AWS, so the same service definition targets either. |
| Bespoke internal platform-engineering scaffolds | Tailored exactly to one company's stack; common in mature orgs | Usually unpublished, un-audited by outsiders, and rarely include a first-class agentic governance layer — that concern barely existed when most internal scaffolds were built. This template treats it as a testable, CI-gated contract from the start. |
The throughline across every row: this template is the only common option that ships a complete infra-to-serving stack and a governed, contract-tested agentic development layer as one coherent, adoptable unit — not a platform to operate, and not a governance story left to prompting.
Multi-IDE Governance¶
Canonical source
A vendor-neutral body store¶
Rules, skills and workflows live once in agentic/ — not named
after any single IDE, so a tool rebrand (this happened once already) can
never strand the source of truth again.
Portable adapters
Devin, Cursor, Claude Code and Codex¶
Devin ingests full bodies, so .devin/ is a generated
byte-for-byte mirror. Cursor, Claude Code and Codex read pointer files.
Both kinds are regenerated by one script and never hand-edited.
Manifest
Cross-surface index¶
The manifest maps rules, skills and workflows across IDE surfaces so the governance model can be validated instead of trusted by memory.
Behavior protocol
AUTO / CONSULT / STOP everywhere¶
Risk class is independent of the assistant being used. Low-risk work can run, ambiguous work asks, and destructive or production-sensitive work stops.
What A Reviewer Should Inspect¶
First scaffold
Can it generate a usable service?¶
A reviewer should be able to scaffold a service, run the local checks and see a coherent FastAPI project with tests and deployment artifacts.
Serving contract
Does inference match training?¶
The service should preserve feature parity, validate inputs and expose readiness based on model loading rather than a superficial health endpoint.
Governance contract
Can agents act safely?¶
The strongest technical signal is whether agent workflows are bounded by rules, manifests, validations and explicit escalation points.
Cloud realism
Are claims scoped honestly?¶
The template distinguishes local validation from real GKE/EKS validation, which keeps the documentation useful without overstating production proof.
What It Shows About Me¶
| Signal | What it means |
|---|---|
| Product thinking | I turned portfolio lessons into a reusable starter system. |
| MLOps discipline | The template treats serving, testing, packaging and deployment as one system. |
| Governance mindset | AI-assisted engineering is bounded by explicit rules, not only prompts. |
| Operational honesty | Cloud validation, cost control and limitations are documented instead of hidden. |
| Documentation taste | The repo is designed so another engineer can adopt, review and improve it. |
Where To Go Next¶
Repository
Template source¶
Start here for the actual scaffold, docs, workflows, rules and release history.
Adoption path
Quick Start¶
Best entry point for generating and validating a new service from the template.
Decision trail
Architecture decisions¶
Review the trade-offs behind the template instead of only reading the final structure.
Portfolio context
Technical evidence¶
See how the template relates to the broader monorepo, cloud evidence and production ML portfolio.