One Entry Point Also Concentrates Failure and Trust

An AI gateway sits between applications and model providers. It gives services a common API for multiple models and centralizes routing, retries, failover, caching, budgets, and logs. For production teams, that can reduce the work of integrating each provider separately and create a shared place to enforce policy.

That abstraction is not free. If the gateway fails, is misconfigured, or has a security problem, the impact may extend beyond one model adapter to every application whose traffic passes through it. Choosing a gateway is therefore not just a question of model coverage. It is also a decision about where to place permissions, observability, and recovery controls.

The Performance Numbers Do Not Share One Ruler

The 2026 guide names Bifrost its self-hosted performance pick and cites figures from its README: 11 microseconds of added latency at 5,000 requests per second on a t3.xlarge, and 59 microseconds on a t3.medium. It also cites a Maxim comparison with LiteLLM reporting P99 latency of 1.68 seconds for Bifrost and 90.72 seconds for LiteLLM at 500 requests per second. Those figures are striking, but the supplied material does not give enough detail to verify the test environment, request mix, or full methodology. The comparison is attributed to Maxim, so it should not be treated as an independent evaluation.

Other projects report numbers with different scopes. Portkey claims latency below one millisecond and routing to more than 1,600 models. LiteLLM reports 8 milliseconds P95 at 1,000 requests per second in its own benchmarks. Added latency, P95, P99, model coverage, and throughput answer different questions. Before comparing them, a team needs to establish what was measured, on what hardware, with what traffic, and whether the figures come from the same test setup.

Open Source, Managed Services, and Existing Platforms Offer Different Control

The products in the guide do not represent interchangeable deployment choices. Bifrost is a Go gateway under the Apache 2.0 license, aimed at teams that need to self-host in a VPC or isolated network. LiteLLM offers a Python SDK and proxy service, with broad coverage across more than 100 providers, but the team must operate the proxy. Portkey uses the MIT license and combines routing with guardrails; the guide lists a free tier of 10,000 logged requests and paid plans starting at $49 per month.

Managed options redistribute operational responsibility and control. The guide describes Cloudflare as edge-oriented with a free core, Vercel as offering zero markup, and OpenRouter as a way to access models. Those labels do not establish that the services are equivalent in governance, private deployment, or incident response. Kong AI Gateway is positioned as bringing LLM, MCP, and A2A traffic into an existing API gateway stack. That may suit teams treating AI traffic as an extension of platform governance, but the material does not provide enough detail to compare its specific deployment constraints or performance.

Governance Turns the Gateway into a Policy Enforcement Point

A gateway’s value is not limited to forwarding a request to the right model. The guide lists virtual keys, budgets, rate limits, access controls, audit logs, guardrails, caching, and failover. Bifrost’s open-source build includes hierarchical budgets and rate limits. LiteLLM supports team spend tracking and RPM and TPM limits. Portkey emphasizes input and output guardrails, along with access control and logging for MCP tool calls.

The more controls a gateway centralizes, the more its configuration becomes an active access and cost policy. Technical leaders should establish whether policies can be scoped separately by team, application, and model; whether logs fit existing observability workflows; who handles false positives and missed detections; and whether traffic can be bypassed or redirected during an incident. The guide mentions Prometheus metrics and OpenTelemetry tracing for Bifrost, and logging integrations including Langfuse and OpenTelemetry for LiteLLM. It does not provide independent evidence of how completely or effectively each product enforces policy.

Ownership and Supply-Chain Events Now Belong in the Evaluation

The guide includes ownership and security history among its production-selection criteria and notes developments that should change the scope of due diligence: Palo Alto Networks completed its acquisition of Portkey on May 29, and Portkey is being folded into Prisma AIRS; Mintlify acquired Helicone, which is now in maintenance mode; and LiteLLM experienced a PyPI supply-chain compromise in March. The material does not explain the scope or follow-up of these events. They are not enough to conclude that a project is currently unsafe or that commercial ownership necessarily makes another option safer.

A more useful response is to turn these signals into concrete checks: how dependencies and release packages are verified, how vulnerabilities are disclosed and fixed, who maintains the project, and how ownership may affect support or the roadmap. Self-hosting teams should also check whether they can pin versions, audit images and dependencies, and switch providers if needed. Teams using managed services should examine data handling, availability commitments, and exit paths. A ranking can narrow the field, but the final choice should be driven by the team’s own workload tests, permission model, and risk tolerance.