Replicate vs Modal vs Baseten: LLM Hosting

Self-hosting open-source LLMs for production in 2026 has moved past the “spin up your own EC2 and fight with CUDA” phase. Three serverless GPU platforms dominate the market for turnkey LLM deployment: Replicate, Modal, and Baseten. All three let you deploy Llama, Mistral, Qwen, or any open-source model with one or two config files, scale to zero when idle, and only pay for GPU-seconds consumed. This article compares them on deployment experience, cold starts, GPU options, pricing, and the specific use cases that make one the right fit.

What each platform is

Replicate: run open-source models with an HTTP API. Popular for image, video, and audio models (Stable Diffusion, Whisper, Flux); also serves LLMs. Model-first UX: find a model, call it, get results. Low developer friction, highest abstraction.

Modal: serverless Python functions on GPU. You write Python code that runs on their infrastructure; they handle GPU allocation, scaling, cold starts. Flexible: build whatever ML workload you want (not just LLM inference).

Baseten: production ML serving platform. Deploy a model as a service with autoscaling, A/B testing, and observability. More opinionated toward operational use cases (feature stores, monitoring, deployment hygiene) than Modal or Replicate.

Deploy an LLM: three approaches

Replicate (public model, zero deploy):

import replicate

output = replicate.run(
    "meta/llama-3.3-70b-instruct",
    input={
        "prompt": "Explain prompt caching in 2 sentences",
        "max_tokens": 200,
    },
)
print("".join(output))

No deploy. Hundreds of popular LLMs already hosted on Replicate’s shared infrastructure. Pay per prediction (per second of GPU time or per 1K tokens depending on the model’s config).

Modal (custom Python, your code):

import modal

image = modal.Image.debian_slim().pip_install("vllm")
app = modal.App("llama-serve")

@app.function(image=image, gpu="A100", scaledown_window=300)
def generate(prompt: str) -> str:
    from vllm import LLM
    llm = LLM(model="meta-llama/Llama-3.3-70B-Instruct")
    return llm.generate(prompt)[0].outputs[0].text

@app.local_entrypoint()
def main():
    print(generate.remote("Hello world"))

modal deploy turns this into a hosted endpoint. Full control over the serving code (vLLM, TGI, custom inference loop, anything you can import).

Baseten (model + config file):

# config.yaml (Truss framework)
model_name: llama-3.3-70b
resources:
  accelerator: A100
  cpu: "4"
  memory: "32Gi"
runtime:
  predict_concurrency: 4

# model/model.py
from vllm import LLM

class Model:
    def __init__(self):
        self.llm = None
    def load(self):
        self.llm = LLM(model="meta-llama/Llama-3.3-70B-Instruct")
    def predict(self, request):
        return {"text": self.llm.generate(request["prompt"])[0].outputs[0].text}

Baseten uses Truss (their open-source framework) as the deployment format. truss push deploys. Includes production observability dashboard, request logs, and canary deployment flows built in.

Pricing (approximate Oct 2026)

Modal is cheapest raw price, Replicate most expensive, Baseten in the middle. Replicate’s price includes their shared infrastructure where popular models are already loaded (lower cold start). Modal and Baseten pricing is for dedicated endpoints.

Cold start comparison

Cold start is the time between a request arriving at an idle endpoint and the model being ready to serve. Hugely important for intermittent workloads.

• Replicate (public shared models): 1-10s. Shared infrastructure keeps popular models hot. Private deployments: 30-120s.
• Modal: 5-30s typical for an LLM container. Supports container warm-up and min_containers for always-warm instances.
• Baseten: 10-60s for a 70B model. Baseten Chains and auto-scaling policies tune the warm-pool size.

For latency-sensitive user-facing chat, cold starts are unacceptable; keep at least 1 instance warm (adds base cost but eliminates cold-start latency). For batch workloads or infrequent inference, scale-to-zero is fine.

Which platform wins by use case

• You want to try a popular open-source model fast: Replicate. Zero deploy, call an API, done.
• You need custom Python logic around the model: Modal. Full code control, not just inference.
• You want production observability out of the box: Baseten. Dashboards, A/B tests, canaries built in.
• Lowest cost at scale: Modal (cheapest GPU-seconds).
• You run image/video models + LLMs: Replicate (strongest multimodal catalog).
• You need SOC 2 for enterprise sales: Baseten (SOC 2 Type II) or Modal enterprise tier.
• You want air-gapped or on-prem: Baseten enterprise supports this; Replicate and Modal do not.
• You need function-level flexibility (one script = one GPU job): Modal. Replicate and Baseten are model-oriented, Modal is code-oriented.

Hidden costs to watch

• Idle warm-pool cost: keeping a GPU warm 24/7 costs $20-200/day depending on GPU type. For low-traffic apps, this swamps the per-request savings vs cold-start acceptance.
• Egress bandwidth: all three charge for outbound data. Large model outputs (long generations, image responses) add up. Check pricing specifics.
• Model storage: Baseten and Modal charge for private model storage. Replicate includes storage for public models in base pricing.
• Overprovisioning: default autoscale settings often spin up too many instances for small bursts. Tune concurrency per instance to maximize GPU utilization.

Decision shortcut

• Prototype: Replicate (zero friction)
• Production + custom code: Modal
• Production + enterprise ops: Baseten
• High-volume at best price: Modal, with warm-pool tuning
• Multimodal workloads: Replicate

None of them lock you in. The underlying inference code (usually vLLM or TGI) runs the same across all three. Switching platforms is a matter of changing the deployment manifest, not rewriting your model.

Common deployment pitfalls

Teams moving an open-source model from laptop to production hit the same handful of surprises. Four worth planning around:

• Cold-start cost on 70B models. Loading a 140GB weights file from object storage into GPU memory takes 60-180 seconds. Scale-to-zero endpoints eat this latency on every first-of-day request. Pre-cached weights (bake into the container image) or keep one instance always warm.
• Overprovisioning on autoscale. Default autoscale policies spin up a new replica on any pending-request count above 1. For bursty traffic this creates 5-10 instances for a 30-second burst, then scales them down after idle timeout. Set a sensible min_concurrency_per_replica (3-10 is typical for LLM workloads) so each GPU handles multiple requests before spawning another.
• Billing surprise from forgotten endpoints. A dev endpoint left running overnight on an A100 ($20/hour) costs $150/day. Set a hard timeout policy or billing alert on dev accounts. All three platforms support auto-scale-to-zero after idle timeout.
• Not setting a request timeout. If the model hangs on a bad prompt, the request can tie up the GPU indefinitely. Set a server-side timeout (TGI’s --max-batch-prefill-tokens, vLLM’s --max-model-len) to kill runaway requests after 60-120 seconds.

Frequently Asked Questions

Glay Eliver

Programmer & Technical Writer at PIES IT Solution

Glay Eliver is a programmer and writer at PIES IT Solution, author of over 600 tutorials at itsourcecode.com. Specializes in JavaScript tutorials, Microsoft Office how-tos (Excel, Word, PowerPoint), and Python error debugging covering ImportError, TypeError, AttributeError, ModuleNotFoundError, and JavaScript ReferenceError. Authored several of the site's highest-traffic Excel and MS Office reference articles.

Expertise: JavaScript · MS Excel · MS Word · MS PowerPoint · Python · Python ImportError · Python TypeError · Python AttributeError · ModuleNotFoundError · JavaScript ReferenceError · Pygame  · View all posts by Glay Eliver →