ML Platform¶
Enterprise multi-project ML platform · monorepo
The second template — where classical ML stops being the whole problem¶
ml-platform is a multi-project ML platform monorepo: one shared substrate — data, features, serving, observability, governance, agentic tooling — reused across projects that differ in kind, not merely in dataset. Tabular ML, time series, deep learning, LLM/RAG and agents, on GCP and AWS.
It is the second template in this lineage, not a successor to the first.
ml-service-template governs classical ML — scikit-learn,
XGBoost, LightGBM, single team, 1–5 models — and is deliberately small enough
to read in an afternoon. ml-platform covers the work that sits outside that
boundary — point-in-time feature retrieval,
lakehouse table formats, orchestration with lineage, GitOps reconciliation,
LLM systems with evaluation gates, and governance artifacts written for an
auditor rather than a reviewer. It consumes the first template through
copier rather than replacing it
(ADR-003).
The Two Templates Are Not The Same Tool¶
Both are multi-cloud and both are governed by ADRs. They answer different questions, and the boundary between them is itself a written decision (ADR-000).
ml-service-template |
ml-platform |
|
|---|---|---|
| Question it answers | "I need governed classical ML in production" | "I need a substrate spanning tabular, DL, LLM and agents" |
| Unit of reuse | A scaffold you copy out and own | A library plus a running service, consumed in-repo |
| Model kinds | Tabular / classical ML only — a stated limit, not an omission | Tabular · time series · deep learning · LLM/RAG · agents |
| Data layer | In-memory DataFrames, Pandera validation | Iceberg lakehouse, BigLake / S3 Tables, DuckDB + Polars, dbt, point-in-time joins with a leakage detector |
| Orchestration | CronJob + GitHub Actions — calibrated for 2–3 models | Airflow 3 + KFP v2, Vertex AI Pipelines and SageMaker Pipelines |
| Deployment | kubectl apply through governed GitHub Actions |
GitOps: ArgoCD + ApplicationSets + Argo Rollouts |
| Observability | Prometheus + Grafana + Evidently | OpenTelemetry + Grafana LGTM (Loki · Tempo · Mimir) + Jaeger |
| Drift | PSI with quantile bins, one tabular detector | Four detectors for four project kinds, one shared contract (ADR-007) |
| LLM / GenAI | out of scope by decision | LiteLLM, prompt registry, semantic cache, guardrails, Langfuse, promptfoo eval gates |
| Anti-pattern model | 38 encoded anti-patterns (D-01→D-38) + 8 audit-standard (Q-01→Q-08) | Technology triage: every tool tiered core / demonstrated / studied / rejected, with the reason |
| Entry cost | Minutes — copier copy a service |
Hours — a monorepo with a six-stage progression |
| Read it in | An afternoon | Not in an afternoon, and that is the trade |
The honest summary: if your models are classical ML, the first template is the right answer and the second is over-engineering. The second earns its complexity only once several projects of different kinds need to share substrate — which is exactly the claim it puts on record to be falsified.
The Claim, Written To Be Falsified¶
A platform's central claim is that shared substrate makes each additional project cheaper. That claim is untestable with one project, so it is written down as a criterion with a test attached:
C1 — a second project reuses ≥3 shared libraries with no fork, verified by a dependency-graph test in CI.
If C1 fails, this is a monorepo of unrelated projects and the platform claim is false. C1 is first testable at Phase 3 — deliberately early, while reversing the decision is still cheap.
Recruiter view
Scope discipline, demonstrated twice¶
Two templates exist because one of them refused to grow. Knowing which problems a tool should not absorb is the judgment this pair is evidence for.
Technical lead view
Claims that can fail a build¶
The claim → gate mapping in docs/governance/quality-gates.md is
itself checked for gaps. A measured number that cannot fail anything is
decoration.
Platform view
Layers, so libs cannot import projects¶
libs/ never imports projects/, projects never
import each other — enforced by a test rather than by review discipline.
Evidence Layers — What A Green Tick Is Allowed To Mean¶
The repository's build state is generated from the filesystem by
scripts/check_implementation_status.py, never maintained by hand, because a
status table a human maintains will drift. Each component also carries the
layer its evidence reaches:
| Layer | What it proves | Count |
|---|---|---|
| L1 | Contract — the test suite passes | 37 |
| L2 | Component — the generator renders, the gate runs, the build completes | 11 |
| L3 | Cluster — it starts and answers in kind |
4 available, not run in CI |
| L4 | Cloud — a real rollout on GKE or EKS | 0 |
L4 is printed at zero on purpose. That taxonomy exists because of a measured failure in this very repository: six Kubernetes overlays rendered green for weeks while their probes pointed at routes the service does not serve, so no pod could ever have reached Ready. A suite passing and a pod answering are not the same claim.
No cloud deployment is claimed here because none has happened. The order is
a constraint the maintainer set explicitly: local validation precedes cloud, and
the first cloud deployment in this lineage is ml-service-template's, not this
repository's.
What Is Built, And What Is Absent¶
Shared libraries — ml-core (drift contract, fairness metrics),
data-contracts (versioned contracts with an explicit compatibility rule),
feature-defs (point-in-time joins plus a leakage detector), llm-core
(prompt registry, semantic cache, guardrails), serving-core (deliberately
near-empty, and the emptiness is declared and tested — one consumer does not
justify a library).
Projects built — demand-forecast (time series, Iceberg ingestion,
panel-aware temporal splitting, expanding-window backtesting),
rag-assistant (LLM/retrieval, provider-fingerprint and cost-per-request
drift), store-assistant (agents, trajectory drift, deterministic policy
gate).
Projects absent — credit-risk, doc-intelligence, agent-ops. Marked
⬜ in the status table, which means absent, not planned.
Drift, per project kind — the inherited machinery was built for one kind, a tabular service. Four kinds get four detectors on one shared contract: PSI with quantile bins for tabular, embedding-space drift for documents, recall-on-frozen-eval-set for retrieval, and tool-use/escalation/policy-gate rejection rates for agents. Plus the failure most easily missed — a provider silently changing the model behind a version alias, where evals degrade with zero code, data or deploy change.
Measurement honesty as a shipped artifact — a semantic retrieval index was built, evaluated against the lexical baseline, tied it (both 15.4% recall@5), failed to clear the 0.05 margin, and no index ships. The number, the diagnosis and what would reverse the decision are published anyway.
Why It Absorbed — And Did Not Replace — agent-local¶
The LLM and agent track named in the charter was already substantially built in
agent-local: a grammar-constrained Tier-0 router, a
deterministic policy gate whose rules are versioned data, a fail-closed tool
capability contract, cross-tier verification and an OWASP-LLM-mapped threat
model.
ml-platform vendored that work in with its git history
(ADR-002) —
core/ became libs/llm-core/, the tienda use-case became
projects/store-assistant/ — because cross-repository coordination costs two
CI configurations, two changelogs and two ADR sets, and a third participant
makes that cost superlinear.
The standalone repository stays live, because its value is not the same
value. ml-platform gives that core one particular, governed use inside a
platform; agent-local is the standalone version of that core, for anyone who
wants it without the platform around it. The core itself is developed here.
Code Review Shortcuts¶
Charter — what it refuses to be (ADR-000) Monorepo topology (ADR-001) Consume, never reimplement (ADR-003) Tooling triage (ADR-004) Drift per project kind (ADR-007) Implementation status (generated) Quality gates — claim → gate Technology inventory Six-stage progression
Where It Sits In The Line¶
ML-MLOps-Portfolio three services, three incidents — where the lessons were paid for
│
├──▶ ml-service-template classical ML · read it in an afternoon
│ │
│ │ consumed via copier (ADR-003), never forked
│ ▼
└──▶ ml-platform the substrate every problem shape shares
▲
│ agent core vendored with history (ADR-002)
│
agent-local business-agnostic LLM agent core, still standalone