Where regulated ML teams get stuck
- Self-managed SCM/CI on infrastructure that can no longer be patched → fails the security audit
- Crown-jewel ML codebase locked to a legacy instance; hard dependency on central IT to operate
- No observability, no model routing/fallback, no compliance tooling — flying partly blind
- Platform work stalled in ticket queues; multi-week outages with no team-owned remediation path
➞
Target State — the platform
- Cross-functional environment the ML team owns end-to-end (ML + MLOps integrated)
- Self-service IDP + GitOps: golden paths, provisioning in minutes, autonomy preserved
- Agentic-native: managed LLMs, model routing/fallback, MCP servers, evaluators
- Observable & compliant by default under a single security policy — not just re-hosted
Target Architecture — layered, with cross-cutting rails
Developer Experience — Self-Service IDPLayer 1
Developer Portalcatalog · docs · ownership
Golden Paths & Scaffoldingminutes to production-ready
Self-Service ProvisioningML workspaces on demand
AI-Assisted Platform CodeClaude Code in the SDLC
GitOps Control PlaneLayer 2
Modern Git Codebasemigrated off legacy SCM
Argo CDdeclarative delivery
ACK / kro / Crossplanecloud resources as CRDs
Kargocanary & dev→prod promotion
CI/CD + DORApipelines & metrics
AI / Agentic LayerLayer 3
Managed LLM GatewayBedrock · Anthropic · OpenAI
Model Router + Fallbackcheapest model that meets rule
MCP Serverstools behind 5 ordered guardrails
RAG (internal)private knowledge
Evaluators / Validatorsevaluate the evaluators
Agent Fleetdomain document processing
Compute & Data FoundationLayer 4
Kubernetes (EKS)platform workloads
Databricksdata & ML pipelines
SageMakertraining / serving
WarehouseSnowflake / equivalent
Object StorageS3
Observability
Every inference is a structured event: guardrails and tool calls as child spans, trace ID stamped into every log line. The trace is the audit trail.
OpenTelemetry → Tempo / Loki / Prometheus → Grafana
Security & Compliance
Single security policy owned with the CISO; lock the audit baseline first, harden incrementally.
Audit baseline · least-privilege · ordered guardrails
Migration Flow — incremental, de-risked
Phase 0 · Now
Stand up & secure
Provision the team-owned environment under the single security policy. Engage the cloud TAM and enterprise credit pool. Lock the security team's must-have baseline.
Phase 1 · ~30 days
Instrument first
Deploy OTel + Grafana/Prometheus across LLM usage and Kubernetes so spend and stability are visible before scaling anything.
Phase 2 · ~60 days
Migrate workloads
Move ML workloads off the legacy instance into the new GitOps codebase incrementally — preserving the autonomy the team has today.
Phase 3 · ~90 days+
Agentic platform
Model routing/fallback, MCP servers, evaluators, and self-service golden paths. Consolidate model hosting behind the managed gateway.