TL;DR
- Put the budget at the model invocation boundary, with a named team, workflow and credential. A dashboard that reports yesterday's usage is not an enforceable limit.
- Portkey fits a managed gateway programme. LiteLLM fits teams prepared to operate the proxy and its spend database. Cloudflare AI Gateway is attractive for an existing Cloudflare estate. OpenRouter is useful when model access and a credit balance should sit behind one API.
- Count retries, fallbacks and cached requests separately. Search, enrichment and other tool bills sit outside the model meter unless you explicitly include them.
- Test concurrent requests and failure behaviour before treating a limit as a hard financial cap. Restrict direct provider credentials so applications cannot bypass the policy.
A model budget needs a boundary
An LLM gateway is an intermediary between an application and model providers. It can centralise model routing, credentials, usage controls and related telemetry. The relevant buying decision is how that intermediary controls model calls; a GTM MCP gateway governs a different boundary around tools and their permissions.
Assign every request to an owner. Use team and workflow identifiers that the application cannot casually invent to obtain a fresh allowance. Keep production, experiments and client work separate. One global budget can stop the whole revenue operation because an experimental research loop consumed the shared pool.
Distinguish rate limits from spend limits. Ten requests per second says nothing about the price of each request. Likewise, a maximum token count does not cover paid search tools or a chain that issues multiple calls. Use both operational limits and a financial policy, then reconcile the gateway's estimates with provider invoices.
Four gateway choices
| Choice | Operating model | Strong reason to evaluate | Principal limitation |
|---|---|---|---|
| Portkey | Managed control layer | Central keys and granular usage policy | Entitlements and contract need checking |
| LiteLLM | Self-managed proxy, with commercial options | Configurable routing and spend controls | Database and proxy reliability become your job |
| Cloudflare AI Gateway | Cloudflare-managed gateway | Existing edge estate and core gateway functions | Price all added features and inference separately |
| OpenRouter | Unified model access with credit controls | Consolidated access and model choice | Account balance is only one part of governance |
Portkey
Portkey's pricing page presents virtual keys, granular budget/rate controls and enterprise access capabilities. The page lists Production at $49 per month with 100,000 recorded logs, while granular budget and rate limits are presented under Enterprise custom pricing. Confirm that the proposed tier includes the required enforcement. Those controls are useful when the buyer wants centrally issued application credentials instead of distributing provider secrets among sales automation builders.
Evaluate it with three real policies: a small experiment budget, a production workflow that must degrade gracefully and an agency client whose spend must remain isolated. Require the commercial proposal to identify gateway request allowance, retention, access controls, support and the exact budget feature included. Do not infer every listed enterprise control is part of a lower-priced plan.
This is a good route for a team wanting managed governance and someone responsible for configuration. It is poor value if every application can still call the provider directly or the organisation has no agreed cost ownership. Model fees and tool fees remain separate from a gateway contract unless its billing arrangement explicitly includes them.
LiteLLM
LiteLLM's budget documentation describes key, user and team controls and warns that database-backed spend tracking is necessary for those controls. Its current documentation also distinguishes cost reservation from routes where only recorded spend can be used. That is a consequential implementation detail: deploying a proxy without its required accounting state can invalidate the financial policy.
Choose it when the engineering team can operate the proxy, database, upgrades and incident response. Keep a versioned configuration and test a database outage. Decide whether the service should stop, allow only a safe limited route or continue under a documented emergency policy.
An open proxy can avoid a managed platform fee, but hosting and staff time are real costs. Obtain a commercial quote where support or enterprise features are required. Do not select it solely because the initial deployment works; spend enforcement depends on the sustained deployment architecture.
Cloudflare AI Gateway
Cloudflare's pricing documentation lists core analytics, caching and rate limiting as free, while added capabilities and inference can have separate charges. Its custom cost configuration is relevant when provider tariffs or cache-token accounting need explicit treatment.
This fits an existing Cloudflare estate looking for a relatively direct gateway insertion. Check the current spend-control feature and account requirements separately from rate limiting; the existence of a core free tier does not prove that every governance requirement is free or included.
The non-fit is a procurement case built on an assumed hard cap without testing its model pricing, authentication and concurrency behaviour. Retain provider invoices as the financial authority and investigate discrepancies between logged estimates and billed usage.
OpenRouter
OpenRouter's limits reference distinguishes balance exhaustion, key credit limits and in-flight spending constraints. These are useful for controlling a workflow's access to a shared model catalogue, but they should not be collapsed into a single “rate limit” response.
Choose it when unified access and model choice simplify a real application requirement. Handle each rejection according to its cause: waiting can help an in-flight limit, while repeatedly retrying an exhausted key cannot create more budget. Keep approved provider/model routes and retention requirements in the procurement review rather than letting automatic routing make every data-handling decision.
Price the actual selected models, credit purchasing conditions and any applicable fees. A consolidated balance does not eliminate the need to attribute usage to teams or to include separate tool calls in the total workflow budget.
Route by approved task
Use a cheaper approved model for a bounded extraction task when its output passes the same fixture as the alternative. Reserve a stronger route for exceptions that need it. Do not let a failed cheap call generate an unlimited ladder of increasingly expensive retries.
Specify maximum attempts, output tokens, elapsed time and total workflow spend. Cache only where reuse is safe. A cache key should represent the model, prompt/policy version, relevant input and tenant boundary. Reusing a customer-specific answer across clients is a data error even if it saves tokens.
Keep the agent monitoring layer for understanding outcomes and failures. The gateway can show model usage, but it does not by itself prove that an account brief was correct or a CRM action was appropriate. Use the ABM Account Scoring Template to define what the workflow should produce before optimising its model cost.
Budget one hundred thousand requests
Assume three teams produce 100,000 requests monthly. Twenty thousand are safe cache hits. The remaining 80,000 average 2,000 input and 500 output tokens, giving 160 million input tokens and 40 million output tokens. At illustrative rates of $1 per million input and $4 per million output, inference costs $320. These are assumed rates, not a named provider quote.
If retries and fallbacks add 10% to those token volumes, inference rises to $352. Add an assumed $100 gateway bill and $200 for paid research tools: total is $652. A dashboard that reports only $352 misses $300 of the workflow cost.
Allocate team budgets by the work they own rather than dividing equally by default. One team may generate brief summaries while another performs deep research. Reserve a shared exception allowance with an explicit approver. Include maximum in-flight exposure: many expensive requests admitted just before a threshold can make recorded spend exceed the nominal limit.
Check enforcement before procurement
Run an inexpensive fixture near a deliberately small test budget. Issue concurrent calls, a retry, a fallback and a request using the wrong team identity. Simulate accounting-store failure and check whether direct provider credentials bypass the gateway. Record the outcomes without claiming production testing that has not occurred.
Prefer a managed option when operating the accounting path is beyond the team. Prefer a self-managed proxy when the control and deployment requirements justify maintaining it. For the GTM coding-agent environment, the same principle applies: the cheapest visible software fee is not necessarily the cheapest reliable operating system.
FAQ
Does a gateway replace provider billing limits?
Use both where available. The gateway attributes and controls application usage; provider-level controls can add another boundary. Reconcile both with invoices, and test concurrent exposure rather than promising that every nominal limit prevents any overshoot.
Are cache hits free?
They may avoid an inference call, but gateway, storage or operation charges can remain. Confirm that the cache is safe for the data and that the billing unit distinguishes a cache hit from a provider request. Include invalidation and policy changes in the design.
Can the gateway cap the whole agent workflow?
Only if every relevant charge is included in its accounting. Enrichment, search, telephony and other tools often have their own meters. Add a workflow ledger or orchestrator policy for total cost rather than calling a model-only budget an agent-wide cap.
Should we automatically fall back to another model?
Only to an approved route with a bounded attempt and cost policy. Different models can have different output contracts and data-handling conditions. Check the output before continuing a consequential action, and stop when the budget or acceptance condition is exhausted.
