All work
ML model marketplace
Model marketplace and GenAI content hub
A marketplace for ML models in the spirit of Replicate, paired with a social hub for generative AI content.
- Client
- Agentic Startup
- Role
- Solution Architect, Delivery Manager, CTO
- Period
- Apr 2024 to Dec 2024
- Sheet
- 2.7 · Rev 2026
Context
- A marketplace of models needs usage accounting, rate limits, billing and multi-tenancy in front of every model.
- Inference cost on GPU had to come down.
What I owned
- The ML Inference API Gateway: usage accounting, rate limiting, billing, multi-tenancy, hybrid inference.
- Research into serverless GPU providers (RunPod, Modal) to bring inference cost down.
Architecture
Clients
- Marketplace (React)
- GenAI content hub
Gateway
- Usage metering
- Rate limiting
- Billing
- Multi-tenancy
Inference
- Own cluster (AWS EKS)
- Serverless GPU: RunPod, Modal
Data
- Kafka
- Debezium
Key decisions
- Hybrid inference, with serverless GPU evaluated to cut cost.
- Metering, rate limits and billing sit in one gateway in front of every model.
Stack
- Python
- FastAPI
- SQLAlchemy
- Taskiq
- Kafka
- Debezium
- AWS EKS
- React
Building something like this?
Open to architecture reviews, fractional CTO work and speaking. I reply within one to three working days.