From April to December 2024 I was solution architect and CTO on a marketplace for ML models, in the spirit of Replicate, with a social hub for generative content next to it. On a marketplace like that, the models have little in common: different frameworks, different hardware, different latency.

The questions in front of every one of them are the same, though. Who is calling? How much are they using? How fast may they call? Who pays, and how much? Those should be answered once, in one place. For us that place was the ML inference gateway: usage accounting, rate limiting, billing, multi-tenancy and hybrid inference behind a single API.

The gateway in front of every model. Simplified.

Count first

Metering is the foundation. Billing, rate limits, quotas and cost reports are all views over the same usage records. Build them separately and every month ends in reconciliation: the invoice says one thing, the limits counted another, the GPU bill a third.

So the gateway’s first job on any request is to tie it to a tenant and start counting. Keep the raw usage events, not only the totals. When a customer disputes a bill, you want to show them the calls behind it.

Tenants share GPUs

On a marketplace many customers share the same hardware. Without limits, one heavy user becomes everyone’s latency problem. Limits belong in the gateway, keyed to the tenant, the same identity that metering and billing use.

This also changes what it costs to add a model. If accounting, limits and billing live in each model service, every new model is a small billing project. In the gateway, a new model gets all three on the day it goes live.

Idle is the expensive part

A rented GPU that waits for traffic costs the same per hour as one that is busy. The useful number is what an hour of actual work costs, and that depends on utilisation far more than on the list price.

GPU busyCost per busy hour
100%$3.00
60%$5.00
25%$12.00
10%$30.00
What an hour of real work costs on a GPU rented at an illustrative $3 an hour.

Who pays for the wait

So the real question in inference pricing is who pays for the waiting. Providers answer it differently, and their pricing pages are worth reading closely.

OptionIdle timeCold start
Replicate, public modelNot billed: you pay for processing time onlyNot billed to you
Replicate, private modelBilled: all the time an instance is onlineBilled: setup time counts
Runpod ServerlessBilled during the idle timeout, then scales to zeroBilled while the worker starts
Runpod, active workersBilled continuously, jobs or notNone: workers stay warm
ModalBilled while a container stays warm, 60 s by defaultAvoidable by keeping containers warm, which is billed
From the providers’ pricing pages and docs, November 2026. Terms change; check before you plan around them.

Each of these is a different setting of the same trade-off. Scale to zero and you pay in cold starts, which for a large model can mean a long wait while weights load. Stay warm and you pay for idle time.

Hybrid inference

Steady traffic is cheapest on capacity you keep running, because its idle share is small. Spiky traffic, and the long tail of models that get a few calls a day, is cheaper on capacity rented by the second. The gateway was built for hybrid inference, our own cluster on AWS EKS next to serverless GPU, and I evaluated Runpod and Modal as the serverless side.

The gateway is what keeps the split invisible. Clients call one API, and the gateway decides where a request runs. Because every call is metered in the same place, the cost of a call stays comparable whichever backend served it.

Same problem, larger

GPU clouds and AI data centres face this at a bigger scale. They sell capacity to many tenants, and their platform layer has to answer the same four questions while keeping utilisation high enough to pay for the hardware.

If you are building one, start with the records. Count every call and tie it to a tenant. Scheduling, limits, pricing and capacity planning all need that data, and it is much harder to add later than to collect from the first day.

The system behind this pieceML model marketplace: Model marketplace and GenAI content hub