The best enterprise AI platform is the one engineers stop noticing. Every account, model call, guardrail, trace, and dollar is handled before anyone asks, and the whole organization builds on it without thinking about it. Thirteen topics, each with why it matters, the target state, and where we are today.
Pick a topic. Inside a topic, arrow keys move between topics and Esc returns here.
The evidence from 2025 and 2026 is unusually consistent: AI pays off in organizations with high-quality internal platforms and fails almost everywhere else.
of enterprise GenAI pilots showed no measurable P&L impact; the differentiator was workflow integration, never model quality.
MIT NANDA 2025
of developers use AI at work. AI amplifies what the organization already is, and platform quality predicts whether usage converts to value.
DORA 2025
of agentic AI projects are predicted to be canceled by end-2027: cost, unclear value, and weak risk controls a platform absorbs.
Gartner 2025
voluntary internal adoption of Spotify's platform in about a year; onboarding cut from 60 days to 20.
Spotify / Backstage
Two more numbers frame the mandate. Shadow AI is the default state: over 90% of workers use personal AI tools regardless of policy, so restrictive governance does not stop usage, it only makes usage invisible. And the METR randomized trial found experienced developers were 19% slower with AI on familiar codebases while believing they were 20% faster, so any adoption program that measures by self-report is measuring vibes.
The platform answer to both is the same. Make the sanctioned path measurably better than the shadow path, and instrument it so you know the truth.
Everything in the other eleven topics is an application of these seven. Each one is testable.
Eight claims a platform leader should be able to state in one breath and defend under cross-examination.
Each layer consumes the one below. The existing estate (engine platform, ML build platform, AI platform) maps cleanly onto the bottom three; the frontier work is layers four and five plus extending the rails to agents.
Solid outline: exists today. Dashed: target state. The three rails apply to every layer, and the frontier is extending them to agents.
The load-bearing decision is that layer three is a choke point on purpose. When every model call in the company flows through one org-controlled gateway, then guardrails, quotas, cost attribution, audit logging, caching, model migration, and regulatory evidence are each solved exactly once. Direct-to-provider calls from application VPCs are blocked at the network layer, which converts policy into physics.
Uber, LinkedIn, and every serious 2026 reference architecture converge on this same shape. The existing centralized API is already this pattern's core, and most of the roadmap is deepening it.
The same platform as a system: how an engineer's work reaches a model or a tool, and which control points it passes on the way. Solid boxes exist today; dashed boxes are target state; the red outline is the gateway choke point. The three rails (identity, observability and FinOps, governance) apply to every box and are drawn on the first tab.
Rendered from ai-platform-map.mmd (Mermaid). Edit the source and re-render to change the map; the explorer loads the SVG next to it.
Every capability an engineer should inherit for free, with honest status. Solid means engineers already take it for granted; partial means it exists but still requires thought; gap means teams are solving it themselves today, which is the definition of platform debt.
| Capability | Status | Target state |
|---|---|---|
| L1 to L2 · Engine and ML build | ||
| Service provisioning and deploy | solid | Account creation to running service in one flow. The benchmark every new capability is measured against. |
| Model serving via Terraform | solid | Config-only deployment of trained models. Extend with autoscaling GPU profiles and canary rollout as defaults. |
| Hardened base images | gap | Minimal, signed, SBOM-emitting containers with risk assessment pre-applied and the guardrail and telemetry clients baked in. |
| Packaging and artifact flow | gap | One-command packaging with Artifactory wired: publish, version, and consume internal libraries and model artifacts without setup. |
| Model registry and cards | gap | Every trained or consumed model registered with owner, version, eval results, and intended use. Generated, never hand-maintained. |
| L3 · Foundation model services | ||
| Centralized model API | solid | Already the single front door. Target: OpenAI-compatible surface so every SDK and framework works unchanged. |
| Guardrails by default | solid | Keep central config; add tiered enforcement (block, flag, log) per route risk and output-side scanning. |
| Telemetry and logging | partial | SignalFx is wired; adopt OpenTelemetry GenAI conventions so every call carries tokens, model, latency, and trace context. |
| Model aliases and version pinning | gap | Apps request chat-default; the platform maps aliases to pinned provider versions. Deprecations become config changes. |
| Prompt and response caching | gap | Provider prompt caching on by default, exact-match response cache, scoped semantic cache as per-route opt-in. |
| Cost attribution and budgets | gap | Virtual keys per team and app; every request tagged; soft alert at 80%, hard stop at 100%; showback from day one. |
| Evaluation as a service | gap | Central eval harness, judge pool, and a registry of core and regression datasets; regressions gate CI. |
| L4 · Agent platform | ||
| Agent registry | gap | Registration is a deploy gate: no production credentials without an owner and cost center. Usage and billing per agent. |
| Agent runtime | gap | Session-isolated, long-running, checkpointed execution with managed memory and sandboxed code and browser harnesses. |
| MCP catalog and gateway | gap | Private subregistry of vetted servers; gateway enforces allowlists, per-user OAuth, audit logs; tool definitions hash-pinned. |
| Agent identity and delegation | gap | Workload attestation, short-lived scoped tokens, explicit delegation chains that policy engines can evaluate. |
| Action-level guardrails and HITL | gap | Tool allowlists per agent identity, policy-as-code on tool calls, human approval for high-impact actions. |
| L5 · Experience layer and rails | ||
| Golden-path scaffolder | gap | One-click "new AI service" and "new agent" templates with gateway client, guardrails, telemetry, identity, and CI pre-wired. |
| Scorecards | gap | Standards continuously evaluated per service, keeping teams on the road without gates or review meetings. |
| Identity, secrets, IAM | solid | Already frictionless for services; extend the same experience to agent identities and provider-key custody. |
| Model inventory / AI BOM | gap | Auto-generated from gateway and registries: the audit-ready answer to "which apps use the affected model?" |
3 of 5 layers solid The next two horizons are L3 depth, then L4.
Three platforms engineers already take for granted are the proof the model works. The job is to extend the same feeling upward.
The highest-leverage component in the whole architecture: a centralized, highly available proxy, deployed natively in the cluster, that every inference call in the company flows through.
Instead of teams managing their own provider keys, all traffic flows through one gateway that speaks the OpenAI-compatible schema every SDK and framework already understands. Because it stands in the request path, every other concern becomes a feature of the road: quotas, guardrails, attribution, caching, model migration, and the audit trail regulators ask for.
The existing centralized API with guardrails by default is already this pattern's core. What remains is depth: aliases, caching, cost attribution, and evaluation, each of which keys off the gateway's position.
Blocking direct-to-provider calls at the network layer turns policy into physics: the sanctioned path is the only path that works.
One request, three stages. The dashed red path is the fallback; the tag on the way out is what keeps evaluation honest (see the fallback dilemma under Evaluation).
An active middleware layer scans every prompt with lightweight NER and regex for secrets, PII, and internal credentials. Matches are masked in transit and restored on return, so the provider never sees the original and the application never notices the substitution.
The same scanner runs on the way out: PII re-leaked in a completion and secrets in generated code are the two output-side failures worth catching centrally.
Prompt-level filtering is necessary and not sufficient. For agents, the enforcement boundary is the tool call, which is why the agent platform adds action-level allowlists on top.
| Option | Where it fits |
|---|---|
| LiteLLMopen source | The default on-ramp: OpenAI-compatible proxy, virtual keys, budgets, and broad provider coverage. Adopt and extend rather than write from scratch. |
| Envoy AI Gatewayopen source · v1.0 | Kubernetes-native and vendor-neutral: token-based rate limiting, quota-aware routing, MCP routing. The infrastructure tier of a two-tier design. |
| Kong / MLflow AI Gatewayopen source | Good fits where the API gateway or MLflow is already the house standard; the same adopt-and-extend stance applies. |
| Portkeymanaged | Deepest AI-native feature set (caching, guardrails, routing) for teams that would rather not operate the layer themselves. |
| Bedrock · Azure APIM · Vertexhyperscaler | Win in single-cloud shops; cross-region inference is the natural fallback target. |
| Uber's in-house gatewayreference | 30 teams, 16M queries a month, and an exact mirror of the OpenAI API so every framework works unchanged. Copy the compatibility decision regardless of vendor. |
A common production pattern is two tiers: an infrastructure gateway for traffic policy plus an AI-native layer for model semantics. Do not write the gateway from scratch.
| Capability | Status | Next step |
|---|---|---|
| Centralized model API | solid | Confirm an OpenAI-compatible surface so frameworks work unchanged. |
| Guardrails by default | solid | Add tiered block / flag / log per route and output-side scanning. |
| Authentication and secrets | partial | Teams never hold raw provider keys: virtual keys issued per app, custody stays in the platform. |
| Model aliases and pinning | gap | Aliases by task tier, pinned versions, shadow and canary routing. |
| Caching | gap | Provider prompt cache on by default; exact-match cache; database primitives ready to provision. |
| Cost attribution | gap | Virtual keys, per-request tagging, budgets with soft and hard limits, showback dashboards. |
Horizon · Now 0 to 2 quarters. Deepening the gateway is the first phase of the roadmap because every later phase keys off it.
Three gateway features that look like optimizations and are actually how the platform absorbs provider churn, cost pressure, and outages on behalf of every team.
The platform ships all three with sane defaults and per-route controls, plus the database primitives (a Redis-class store and a vector store) as attached services that need no manual provisioning.
Order the layers by certainty: deterministic first, probabilistic last, and opt-in where the hit rate has to earn its keep.
Apps request chat-default, extract-cheap, or reason-deep. The platform maps each alias to a pinned provider model version.
A provider deprecation becomes a platform config change instead of forty team migrations, and the swap is validated against mirrored production traffic before any team sees it.
Provider churn is a certainty; enterprise share shifted dramatically between 2023 and 2026. Aliases make churn a config change.
During a primary model outage or a burst of 429s, the gateway silently routes payloads to a secondary model: cross-region Bedrock, or a self-hosted vLLM deployment, with no application code change. Three failure classes deserve three different fallbacks: transient errors retry on the same tier, content-policy rejections route to a model with different filters, and context overflow routes to a longer-context model or a summarizing pre-step.
What a fallback must never do is pretend. Every response carries the model that actually answered, because a silent swap corrupts every quality baseline downstream. That is the fallback evaluation dilemma, and it has a clean fix: see Evaluation as a service, the fallback dilemma.
| Failure | Fallback |
|---|---|
| Outage or 429 | Cross-region Bedrock first, self-hosted vLLM second; per-provider circuit breakers decide when to trip. |
| Content-policy rejection | A model with different filter behavior, flagged on the span so the route owner sees the pattern. |
| Context overflow | A longer-context model, or a summarizing pre-step, rather than a truncated prompt. |
| Capability | Status | Next step |
|---|---|---|
| Prompt and response caching | gap | The identified roadmap item: caching plus a ready-to-spin-up database. Start with provider prompt cache and exact match. |
| Model aliases and pinning | gap | Define the task tiers, map aliases, pin versions, enable shadow routing. |
| Fallback routing | gap | A gateway policy with provenance tagging on every response; the fallback model must pass the baseline gateway test first. |
Horizon · Now 0 to 2 quarters, alongside cost attribution; all three are gateway configuration once the gateway owns the request path.
Traditional monitoring cannot detect hallucinations or semantic regressions. The platform owns the harness, the judges, and the dataset registry; teams own their datasets.
of organizations run offline evaluations at all. Dashboards are table stakes; evals are the differentiator and where reliability is actually won.
industry surveys, 2025 to 2026
Continuous evaluation is a control plane, not a dashboard: sample production traces, route them to an evaluation cluster, and act on the deltas.
Teams write their datasets. The platform owns everything else.
A silent fallback reads as a regression in the blended baseline. Hard-tagged provenance turns it into two clean series and one honest system view.
gen_ai.response.model on the OpenTelemetry span, preserving the requested alias separately in gen_ai.request.model. No application can forget to do it.| Capability | Status | Next step |
|---|---|---|
| Evaluation as a service | gap | Eval harness v1: dataset registry, judge pool, CI gate on the highest-traffic routes. |
| Provenance on spans | gap | Ships with the gateway's OTel adoption; a one-line attribute with outsized value. |
| Production sampling | gap | Sample traces into an evaluation cluster; promote failures into regression sets weekly. |
Horizon · Now 0 to 2 quarters for v1. For the mental model behind core and regression datasets, judges, and reading a run, see the Agent Evaluation walkthrough.
Between 2025 and 2026 the answer consolidated into a recognizable stack. The strategic move is clear: buy the runtime primitives, standardize the protocols, and build the thin control plane that encodes your governance.
Agents are first-class workloads now. The gaps the team has already named (an agent-core style registry, an MCP catalog, runtime management) are this layer.
The registry is the org's agent control plane and the correct first build, because everything else keys off it. Identity issuance requires a registry entry. Gateway virtual keys tie to a registry entry and a cost center. Billing joins model spend (metered at the gateway) with compute (billed by the runtime), so usage and cost are recognized per agent from the first day it runs.
Registration as a deploy gate means "what agents exist, who owns them, what can they do, and what do they cost" is always answerable. There is no cross-vendor registry standard yet: A2A defines the agent-card descriptor, not the registry API. A thin internal registry that exports A2A cards avoids lock-in to any vendor's schema.
com.enterprise.support/refund-assistant · owner: support-eng · cost center 4410chat-default · tools orders/lookup, refunds/status · no write tools without approvalOne registry entry. Every other control plane reads from it: identity, the gateway, billing, evals, and the inventory.
Per-session microVM isolation, checkpointed long-running execution, and sandboxed code-execution and browser harnesses are now vendor primitives: AWS Bedrock AgentCore (sessions up to eight hours, framework-agnostic), Microsoft Foundry Agent Service, Google's agent stack, or self-hosted LangGraph Platform where data residency demands it.
Expect platform overhead around 1.15 to 1.4 times model spend. Building this in-house is a multi-quarter project with no differentiation.
The differentiation is in the policy layer above the runtime. Nobody wins by operating their own microVM fleet.
Standardized memory is an attached service requiring zero manual provisioning: ephemeral session memory on a Redis-class store, and long-term semantic memory on a managed vector database (Milvus or Qdrant class, or the cloud's serverless vector store). The runtime exposes short-term, long-term, and procedural memory through one interface so teams never build their own persistence.
| Memory | Backing store | Owned by |
|---|---|---|
| Short-term (session) | Redis-class, TTL per session, checkpointed by the runtime | Platform |
| Long-term (semantic) | Managed vector store; embeddings versioned so a model upgrade re-indexes rather than silently degrades | Platform, with per-agent namespaces |
| Procedural (skills and tools) | The tool catalog and prompt registry, versioned and hash-pinned | Platform |
| Feature and vector stores for custom models | The ML build platform's gap: same primitives, exposed through Terraform | Platform |
| Capability | Status | Next step |
|---|---|---|
| Agent registry | gap | Thin internal registry as a deploy gate; usage and billing per agent; A2A card export. |
| Agent runtime | gap | Buy the isolation primitive; wire it to the registry, identity, and gateway. |
| Memory service | gap | Redis and vector store as attached services with per-agent namespaces. |
| Action-level guardrails and HITL | gap | Tool allowlists per agent identity; human approval gates for high-impact actions. |
Horizon · Next 2 to 4 quarters. The registry comes first; the runtime, memory, and catalog plug into it.
MCP won the tool-integration war. The enterprise problems are now catalog, auth, and supply chain, and all three resolve to the same question: which identity is calling which tool, on whose behalf?
Agents are only as powerful as the tools they can execute safely. The Model Context Protocol standardizes the interface and decouples orchestration logic from tool execution, which is exactly what lets the platform put one control point between every agent and every tool.
Where tools are published, discovered, reviewed, and versioned. Consume or register, self-serve.
Per-user OAuth through the gateway; never a shared service account, which recreates the confused-deputy problem at scale.
Tool descriptions are an injection surface. Hash-pin definitions at approval; scan for tool poisoning at registration.
A tool allowlist is only as strong as the identity it binds to. Catalog, gateway, and identity are one design.
A centralized metadata catalog where teams publish MCP servers using reverse-domain namespacing, consuming the official MCP Registry's schema so nothing is bespoke. Agentic frameworks discover and subscribe to tools programmatically; virtual servers expose role-scoped subsets of the catalog.
com.enterprise.hr/pto-querycom.enterprise the organization · hr the owning domain · pto-query the toolhr-assistant exposes this tool read-only; hr-admin adds pto-approveOne catalog entry. Reverse-domain names make ownership and discovery unambiguous across hundreds of servers.
RFC 8693 token exchange at every hop; the delegation chain rides in the token so the policy engine sees the whole lineage.
Agents are neither users nor classic service accounts; they are a third identity class. Each agent gets its own identity with workload attestation (SPIFFE is the reference), short-lived scoped tokens via OAuth token exchange, and an explicit delegation chain recorded in the token. Federated identity means the MCP server authenticates the session-scoped OIDC token directly, so data access governance is enforced where the data lives.
Secrets follow the token-vault pattern: agents hold no long-lived credentials, ever. This rail is what makes action-level guardrails enforceable.
| Capability | Status | Next step |
|---|---|---|
| Identity, secrets, IAM for services | solid | Service accounts, secret management, and lifecycle already frictionless. Extend the same experience to agents. |
| MCP catalog and gateway | gap | The named roadmap item: consume or register. Private subregistry, per-user OAuth, hash-pinned tools, audit logs. |
| Agent identity and delegation | gap | Workload attestation, token vault, delegation chains; no static secrets. |
Horizon · Next 2 to 4 quarters, in step with the registry, because identity issuance requires a registry entry.
The experience layer is where "taken for granted" is actually manufactured. Engineers adopt the platform when it is the path of least resistance, so the unit of delivery is a scaffolded template.
The highest compliment an infrastructure or platform engineer can receive is that their work is completely taken for granted. When a platform is built right, developers do not think about risk compliance, IAM roles, or cluster provisioning; they write code and ship features. The strategy shifts from providing APIs to building an invisible exoskeleton around the engineer's code that handles security, cost, caching, and model routing automatically.
The paved road is the easiest road, never a mandate. Adoption is won on merit.
Self-service is the bar for done. A capability that requires a ticket is not shipped.
Design test. Can an engineer who has never spoken to the platform team go from idea to a traced, governed, cost-attributed deployment before lunch? If it needs a meeting, a ticket, or a wiki spelunk, it is not done.
The hardened image underneath is cleared once; security review happens per image, not per team.
Continuously evaluate every service against the standards: evals present, costs tagged, guardrails on, base image current. Drift becomes visible without a single review meeting.
Teams can leave the road and own the consequences. Their off-road pain is the platform's best product-discovery signal; treat it as a backlog, not a violation.
Anyone can contribute a template, an MCP server, or an eval rubric; the platform team curates. Users become co-owners, and golden paths are embedded in onboarding so the paved road is the first road every new engineer learns.
Treat the platform like an internal product, with an internal developer portal (Backstage-class) as its storefront. Engineers see the model catalog with latency and cost per alias, register agents and MCP tools, grab boilerplates, and read the scorecards, all in one place. Solve the systemic barriers there, and engineers focus entirely on applying AI to business logic.
voluntary adoption at Spotify, with onboarding cut from 60 days to 20, once golden paths lived in the portal and in onboarding.
Spotify / Backstage
where most Backstage adopters stall, because the portal shipped without the product work: no paved road worth finding, no marketing, no scorecards.
industry reports
The portal is the shop window. The golden paths behind it are the product.
| Capability | Status | Next step |
|---|---|---|
| Service provisioning and deploy | solid | Already the benchmark. Keep it that way. |
| Hardened base images | gap | Standardized images with risk assessment pre-applied: the named roadmap item. |
| Packaging and Artifactory flow | gap | One-command publish and consume. |
| Golden-path scaffolder | gap | "New AI service" and "new agent" templates with everything pre-wired. |
| Scorecards | gap | Standards evaluated continuously; no review meetings. |
| Developer portal | gap | Catalog, scaffolder, scorecards, docs in one place. |
Horizon · Now → Later Images and packaging now; the scaffolder next; scorecards and the portal once there is a road worth scoring.
Everything else in this explorer describes the platform from the architect's seat. This topic walks the same platform as the engineer meets it: what they do, what they see, and what happens invisibly on their behalf.
The benchmark is the design test from the golden-paths topic: idea to a traced, governed, cost-attributed deployment before lunch. Here is the hour, minute by minute, for an engineer adding a refund assistant to an existing service.
chat-default, extract-cheap, reason-deep) with latency, cost per thousand tokens, and the risk tier each is cleared for. No provider names, no keys, no approval form.platform new ai-service refund-assistant. The repo arrives with the gateway client, guardrails middleware, OpenTelemetry, workload identity, a CI pipeline with an eval gate and image scan, and a Dockerfile on the golden image. A registry entry already exists with the engineer as owner and the team's cost center.chat-default. The virtual key was issued at scaffold time; the gateway authenticates, checks the budget, runs guardrails, and routes. The response returns in 0.4 seconds and the trace is already on the default dashboard with its cost.evals/core.jsonl from real tickets. platform eval runs the harness with the platform's judges and prints a scorecard. The CI gate now fails any change that regresses those cases.git push. The pipeline scans the image, runs the evals, and deploys to the cluster. A URL comes back; the scorecard reads evals present, costs tagged, guardrails on, image current.com.enterprise.orders/lookup. The registry records the grant, the agent's identity gets a scoped token for it, and the audit log starts on the first call. Still no ticket.The first AI call as a sequence: one engineer action per step, and everything between the portal and the provider handled by the platform. Rendered from ai-platform-onboarding.mmd.
Under an hour, no ticket, no human. That is the number the whole platform is measured by: time to first AI call.
The same hour as the decisions the engineer actually makes. Diamonds are the only places a human chooses; everything between them is the platform. The tier-1 branch is the whole point: for most use cases there is no review box on the path at all, and the two loops (eval gate, scorecard) replace review meetings with signals.
Activity flow for onboarding. Solid: exists today. Dashed: target state. Red outline: the first gateway call, the moment the time-to-first-AI-call clock stops. Rendered from ai-platform-activity.mmd.
Count the diamonds: four decisions, zero tickets. If a sixth box appears that says "wait for a person", that box is the next roadmap item.
The invisible exoskeleton, made visible once. Every row on the left is a thing the engineer does; every row on the right is what happens without them asking.
| The engineer does | The platform does, invisibly |
|---|---|
| Picks a golden path in the portal | Shows only aliases cleared for the engineer's risk tier; hides provider names, keys, and pricing. |
| Runs the scaffolder | Creates the repo from the template, the registry entry, the virtual key, the workload identity and IAM role, the secrets binding, the CI pipeline, and the default dashboards. |
| Calls an alias | Authenticates, checks the budget, masks PII, checks the cache, resolves the alias to a pinned version, fails over on a 429, restores masked values, emits a span with cost and model.actual. |
| Writes eval cases | Stores and versions them, runs the judge pool, gates CI on a statistically meaningful delta, promotes failing production traces into the regression set weekly. |
| Pushes to main | Scans the golden image, runs the evals, deploys, registers the route in the inventory, updates the scorecard, starts the budget meter. |
| Subscribes to a tool | Checks the allowlist against the agent's identity, issues scoped tokens per user, hash-pins the tool definition, writes the audit log. |
| Reads the dashboard | Joins gateway spans, runtime compute, and cache savings into one cost-per-task number per route, with an anomaly alert for runaway agents. |
service: refund-assistant · owner: support-eng · cost_center: 4410 · risk_tier: 1 · model: chat-default · budget_usd: 400core.jsonl (15 cases) · regression.jsonl (empty on day one; the platform fills it from production failures)FROM golden/python-ai:1.4; risk assessment, certificates, telemetry agent, and guardrail client already insideThe scaffolded repo. One small declaration file is the engineer's entire contact surface with the platform.
The same hour against the current estate, step by step. The middle column holds only what the platform provides today; where there is no platform path yet, it says so rather than guessing how teams cope.
| Step | Platform path today | Target | Status |
|---|---|---|---|
| Discover | No platform path yet | Portal with golden paths and a model catalog of aliases, latency, cost, and risk tier | gap |
| Account and secrets | Service account creation, secret management, and identity and access are already easy across the lifecycle | Issued by the scaffolder as a side effect; never requested | solid |
| Model access | One organization-controlled API with risk assessment and guardrails applied by default | Same, plus a virtual key with a budget and aliases instead of model names | partial |
| Start the repo | No standardized base image or packaging flow yet (both on the gap list) | One command: template, golden image with risk assessment pre-applied, Artifactory wired, CI pre-wired | gap |
| Deploy | Engine platform: Kubernetes-backed, from AWS account creation to a deployed service with minimal friction | Unchanged; this is the benchmark | solid |
| Serve a custom model | ML build platform: a Terraform configuration is all it takes; training stays free-form | Unchanged, plus a registry entry and model card generated from the build | solid |
| Observe | Telemetry and SignalFx set up automatically, no per-team logging setup | Plus OTel GenAI spans: tokens, cost, model.actual per call | partial |
| Evaluate | No platform path yet | Central harness, judges, CI gate; teams write cases only | gap |
| Build an agent | No registry, runtime management, or MCP catalog yet (all on the gap list) | Registry as deploy gate, managed runtime, MCP catalog subscribe | gap |
| Cache responses | No caching layer or ready-to-spin-up database yet (on the gap list) | Provider and exact-match caching on by default; the database one click away | gap |
The steps marked solid are the proof: engineers already forget they happen. The gaps are the steps with no road yet.
DevEx is the product surface of the platform. If the first hour needs a human, the platform is not done.
| Capability | Status | Next step |
|---|---|---|
| Provisioning, serving, model API | solid | Already invisible. Protect them as the benchmark for every new step. |
| Portal and golden-path docs | gap | Start with two written golden paths and a model catalog page before any portal software. |
| Scaffolder | gap | "New AI service" template first; "new agent" once the registry exists. |
| Virtual keys and budgets | gap | Issued by the scaffolder; the first thing the gateway deepening delivers. |
| Eval harness | gap | v1 with the dataset registry and CI gate. |
| Onboarding telemetry | gap | Measure time to first AI call from gateway logs today, before anything else ships. |
Horizon · Now → Next The measurement and the written golden paths cost nothing and start now; the scaffolder lands with the agent layer.
Governance succeeds as a platform capability and fails as a committee. The design goal: a fast lane by default, real scrutiny where the stakes are real, and the evidence generated as a by-product.
Governance is a fast lane, not a gate. The alternative is a review board engineers route around.
Lane width is volume. The majority of use cases live in tier 1 and never wait for a person; each gate further down is a real check, run in parallel rather than in sequence.
| Tier | Path |
|---|---|
| 1 · Low | Self-serve on the paved road. No review: the platform defaults are the approval. |
| 2 · Moderate | Standard eval bar plus a time-boxed SLA review (days, not weeks); security and legal in parallel, never sequential. |
| 3 · High | Independent validation, human oversight assignment, enhanced logging; pre-approved patterns fast-track repeats. |
| 4 · Critical | Executive sign-off and external audit. Rare by design. |
The EU AI Act's transparency obligations became enforceable in August 2026 (AI disclosure in chat surfaces, machine-readable labeling of generated content), with high-risk obligations following from late 2027: assigned human oversight, input-data control, automatic log retention of at least six months, and incident reporting. NIST AI RMF supplies the internal control vocabulary; ISO/IEC 42001 is the certifiable management system increasingly demanded in procurement.
Teams inherit all of it by building on the road. Compliance is a property of the platform, not a project for each team.
The banking discipline maps well: inventory everything, validate proportionally to impact, and treat provider model updates as change-management events, which contractual version pinning makes possible.
Which models, versions, datasets, prompts, and consuming apps exist, with owner and risk tier, generated automatically from the gateway and registries. A hand-maintained inventory is stale on day one.
The inventory is also the incident-response index: when a provider ships a regression or a jailbreak, "which apps are affected" is one query.
Model cards and the AI BOM are outputs of the platform, never documents someone remembers to update.
| Capability | Status | Next step |
|---|---|---|
| Guardrails and risk assessment at the API | solid | Already applied by default. Add tiering so enforcement matches route risk. |
| Risk-tier fast lanes | gap | Define the four tiers; make tier 1 zero-review; pre-approve repeat patterns. |
| Model inventory / AI BOM | gap | Generate from the gateway and registries once they exist. |
| Policy-as-code | gap | Publication review for tools and templates expressed as policy, evaluated automatically. |
Horizon · Later 4 to 8 quarters for the self-governing state; tiering itself can start as soon as the gateway carries route metadata.
Enterprise AI needs structured data and an asynchronous control plane. The data plane auto-instruments every call; the control plane turns those spans into cost, quality, and adoption truth.
SignalFx and logging are already wired automatically, which is exactly the "taken for granted" bar. What is missing is the AI-specific schema: which model answered, how many tokens in and out, what it cost, and which user, agent, and tool were involved. Without that, cost attribution, evaluation, and adoption measurement are all estimates.
Auto-instrumentation injects distributed trace headers and the GenAI attributes on every call, from the template and the gateway, with no per-team setup.
Asynchronous rollups: showback by cost center, anomaly detection for runaway agents, sampled traces routed to evaluation.
Principle 7 applies: measure honestly, including the costs, and never by self-report.
Standardize on the OpenTelemetry GenAI semantic conventions so every span is portable across SignalFx, Langfuse, Arize Phoenix, or whatever comes next. Specialized observability and evaluation platforms are bought, not built, and they all read this schema.
chat-default (the alias the app asked for)provider-b/model-x-2026-06 fallback (the actual responding model; the requested alias is preserved separately)gen_ai.usage.output_tokens 318 · cache read 3,900["stop"]u-3381 · agent com.enterprise.support/refund-assistant · cost center 4410One span. Cost, quality partitioning, chargeback, and the audit trail all read from these attributes.
Cost visibility precedes cost accountability. FinOps for AI is its own discipline now.
| Metric | What it measures, and the bar |
|---|---|
| time_to_first_ai_call | Minutes from "team wants a model" to the first authenticated, logged, budgeted call. Target: under an hour, no human involved. |
| voluntary_adoption | Share of AI workloads on the paved road with no mandate. Spotify's 99% is the existence proof; trend beats absolute. |
| shadow_ai_rate | Share of model traffic bypassing the gateway. The honest inverse of adoption; drive toward zero by being better, not by blocking. |
| eval_coverage | Share of production routes with core and regression datasets and CI gates. Only about 52% of organizations run offline evals. |
| cost_per_task | Attributed unit economics per team, app, and agent, including cache savings. |
| quality_counter_metrics | Bugs per developer, incidents per PR, review latency, PRs merged unreviewed. Velocity gains can ship a quality tax; watch both sides. |
| platform_nps / dxi | Quarterly internal survey plus a DX-style index. Satisfaction is the leading indicator of adoption. |
| consumer_dora | Deploy frequency, lead time, change-failure rate of consuming teams. The platform's success shows up in its customers' numbers. |
Distrust self-report (METR's minus 19% actual versus plus 20% perceived) and acceptance-rate vanity metrics. Pre-register success thresholds before pilots.
| Capability | Status | Next step |
|---|---|---|
| Telemetry and SignalFx | partial | Wired automatically today; adopt the OTel GenAI attributes so spans carry model, tokens, and cost. |
| Cost attribution and budgets | gap | Virtual keys, per-trace cost, budgets, showback dashboards. |
| Per-agent usage and billing | gap | Follows from the registry plus gateway tagging; the named "usage and billing recognition quickly" goal. |
Horizon · Now 0 to 2 quarters. Attribution is the first thing the gateway deepening delivers.
Team shape, funding, adoption as a program, and the decision frameworks a platform leader is asked to defend: build versus buy, model strategy, vendor management, portfolio discipline.
A small central platform team owning the paved road, with federated contribution via inner-source. New capabilities incubate in collaboration mode with one or two pilot teams, then harden into X-as-a-Service for everyone.
Enabling work (embedded rotations, office hours, coaching) is explicitly temporary per engagement. That is how the team avoids becoming a ticket queue that scales linearly with adopters.
Product-funded, not project-funded, with the roadmap defended by a platform PM. Cost flows through showback first and chargeback at maturity.
Internal marketing is engineering work: publish adoption numbers and wins quarterly, demo at all-hands, embed golden paths in onboarding. Platforms that skip this stall at 10% adoption even with superior technology.
Organic uptake plateaus around 15 to 20% without enablement. The proven package has four parts, and champions programs roughly double adoption versus top-down rollout.
| Capability | Stance | Rationale and selection |
|---|---|---|
| Foundation models | Buy (rent) | Commoditized. Managed services (Bedrock-class) for tier-1 models; fine-tune or self-host only for demonstrated scale, latency, sovereignty, or unit-economics cases. Self-hosting pencils out only past hundreds of millions of tokens a month. |
| Model gateway | Adopt and extend | Never from scratch. LiteLLM, Envoy AI Gateway, Kong, or MLflow AI Gateway, extended with the org's policy. |
| RAG and vector stores | Buy (managed) | Database maintenance is toil. Managed vector stores (cloud serverless or a managed Milvus / Qdrant / Pinecone class). |
| Agent runtime | Buy | AgentCore-class isolation and memory, or self-hosted LangGraph Platform where residency demands it. |
| Agentic frameworks | Build thin wrappers | Raw frameworks underneath; opinionated enterprise wrappers on top that hook into auth, the registry, and telemetry. |
| Evals and observability | Buy | Langfuse, Arize Phoenix, and peers, integrated through the existing OpenTelemetry pipeline. |
| Registry, risk tiers, cost model | Build | The thin control plane that encodes how the organization decides. No vendor can ship your policy. |
MIT's data: purchased solutions succeeded about 67% of the time versus 33% for internal builds. Buy the commodity horizontals; build the thin layer that differentiates. And keep the forgotten branch honest: many "AI" problems are classification or rules, and a non-LLM solution is cheaper and more reliable.
Each frame compresses to a rule with a number behind it. MIT's finding that back-office automation out-earns the sales and marketing use cases that absorb most budget is the standing argument for central portfolio review.
Deepen the gateway, stand up the agent layer, then make the platform self-governing. Every item is a gap named in the capability map, placed where it has the most leverage.
Deepen the gateway. Everything here is configuration of a control point that already exists, which is why it goes first.
Stand up the agent layer. The registry comes first because identity, billing, and the catalog all key off it.
Make it self-governing. The platform answers the regulator, the CFO, and the incident commander without anyone assembling a document.
Research compiled August 2026. Vendor-published figures are treated as directional.