Skip to content
All work

ML model marketplace

Model marketplace and GenAI content hub

A marketplace for ML models in the spirit of Replicate, paired with a social hub for generative AI content.

Client
Agentic Startup
Role
Solution Architect, Delivery Manager, CTO
Period
Apr 2024 to Dec 2024
Sheet
2.7 · Rev 2026

Context

  • A marketplace of models needs usage accounting, rate limits, billing and multi-tenancy in front of every model.
  • Inference cost on GPU had to come down.

What I owned

  • The ML Inference API Gateway: usage accounting, rate limiting, billing, multi-tenancy, hybrid inference.
  • Research into serverless GPU providers (RunPod, Modal) to bring inference cost down.

Architecture

Clients

  • Marketplace (React)
  • GenAI content hub

Gateway

  • Usage metering
  • Rate limiting
  • Billing
  • Multi-tenancy

Inference

  • Own cluster (AWS EKS)
  • Serverless GPU: RunPod, Modal

Data

  • Kafka
  • Debezium
Fig. 2.7. Simplified layer view, redrawn for this site from the project description.

Key decisions

  • Hybrid inference, with serverless GPU evaluated to cut cost.
  • Metering, rate limits and billing sit in one gateway in front of every model.

Stack

  • Python
  • FastAPI
  • SQLAlchemy
  • Taskiq
  • Kafka
  • Debezium
  • AWS EKS
  • React
NFT marketplaceNext sheet

Building something like this?

Open to architecture reviews, fractional CTO work and speaking. I reply within one to three working days.