From April to December 2024 I was solution architect and CTO on a marketplace for ML models, in the spirit of Replicate, with a social hub for generative content next to it. On a marketplace like that, the models have little in common: different frameworks, different hardware, different latency.
The questions in front of every one of them are the same, though. Who is calling? How much are they using? How fast may they call? Who pays, and how much? Those should be answered once, in one place. For us that place was the ML inference gateway: usage accounting, rate limiting, billing, multi-tenancy and hybrid inference behind a single API.
Count first
Metering is the foundation. Billing, rate limits, quotas and cost reports are all views over the same usage records. Build them separately and every month ends in reconciliation: the invoice says one thing, the limits counted another, the GPU bill a third.
So the gateway’s first job on any request is to tie it to a tenant and start counting. Keep the raw usage events, not only the totals. When a customer disputes a bill, you want to show them the calls behind it.
Tenants share GPUs
On a marketplace many customers share the same hardware. Without limits, one heavy user becomes everyone’s latency problem. Limits belong in the gateway, keyed to the tenant, the same identity that metering and billing use.
This also changes what it costs to add a model. If accounting, limits and billing live in each model service, every new model is a small billing project. In the gateway, a new model gets all three on the day it goes live.
Idle is the expensive part
A rented GPU that waits for traffic costs the same per hour as one that is busy. The useful number is what an hour of actual work costs, and that depends on utilisation far more than on the list price.
| GPU busy | Cost per busy hour |
|---|---|
| 100% | $3.00 |
| 60% | $5.00 |
| 25% | $12.00 |
| 10% | $30.00 |
Who pays for the wait
So the real question in inference pricing is who pays for the waiting. Providers answer it differently, and their pricing pages are worth reading closely.
| Option | Idle time | Cold start |
|---|---|---|
| Replicate, public model | Not billed: you pay for processing time only | Not billed to you |
| Replicate, private model | Billed: all the time an instance is online | Billed: setup time counts |
| Runpod Serverless | Billed during the idle timeout, then scales to zero | Billed while the worker starts |
| Runpod, active workers | Billed continuously, jobs or not | None: workers stay warm |
| Modal | Billed while a container stays warm, 60 s by default | Avoidable by keeping containers warm, which is billed |
Each of these is a different setting of the same trade-off. Scale to zero and you pay in cold starts, which for a large model can mean a long wait while weights load. Stay warm and you pay for idle time.
Hybrid inference
Steady traffic is cheapest on capacity you keep running, because its idle share is small. Spiky traffic, and the long tail of models that get a few calls a day, is cheaper on capacity rented by the second. The gateway was built for hybrid inference, our own cluster on AWS EKS next to serverless GPU, and I evaluated Runpod and Modal as the serverless side.
The gateway is what keeps the split invisible. Clients call one API, and the gateway decides where a request runs. Because every call is metered in the same place, the cost of a call stays comparable whichever backend served it.
Same problem, larger
GPU clouds and AI data centres face this at a bigger scale. They sell capacity to many tenants, and their platform layer has to answer the same four questions while keeping utilisation high enough to pay for the hardware.
If you are building one, start with the records. Count every call and tie it to a tenant. Scheduling, limits, pricing and capacity planning all need that data, and it is much harder to add later than to collect from the first day.
The system behind this pieceML model marketplace: Model marketplace and GenAI content hub