Configuring Dedicated Model Inference

Configuring Dedicated Model Inference Webflow Analyze/Optimize tracking bridge -->

💰 Announcing our Series C. Intelligence should be abundant, not expensive →

🤝 Together AI & Y Combinator announce partnership to deliver the first dedicated YC GPU cluster →

⚡ On-demand B200s now available on Together GPU Clusters →

🚀 Now serving MiniMax-M3 for efficient inference →

All blog posts

Inference

Published 7/29/2026

Configuring Dedicated Model Inference

How endpoints, deployments, and configs fit together, and why routing follows capacity, not fixed percentages.

Authors

Mitali Meratwal, Zain Hasan

Table of contents

40+ Models Chosen for Production...40+ Models Chosen for Production...40+ Models Chosen for Production...

Summary

Dedicated Model Inference on the Together AI platform consists of three parts: the endpoint (a stable name you or your clients call), deployments (specific model + hardware combinations running replicas behind it), and configs (recipes for how a model runs). A capacity-aware traffic split ties these three entities together. This architecture enables various other features such as rollouts, A/B tests, shadow experiments, zero-downtime changes possible. Below we'll show how capacity-aware routing works against a live endpoint with measured traffic.

How the resource model works A config is a recipe that specifies the engine, the GPU type and count as well as the parallelism and optimization profile (throughput, latency, balanced). Configs are immutable and every config has an ID like cr_... . Your deployment always points at exactly the config that was tested. A deployment binds one model (at a specific revision) to one config, gives it an autoscaling policy , and runs replicas. Deployments are disposable on purpose and creating and destroying should be considered routine. An endpoint is a fixed identity: a qualified name ( / ) that your applications pass as the model parameter in the ordinary inference API. The endpoint has a traffic split which is used to determine request routing over its deployments.

One easy way to understand this is by referring to the ID, every ID in the system tells you what it is by prefix, which makes logs and scripts self-documenting: proj_ (project), ml_ (model), cr_ (config revision), endpoint_ , dep_ (deployment), rol_ (rollout).

Every advanced operation in the platform is just "add a deployment, and assign it a traffic routing weight" An A/B test is deployments with cohort assignments. A shadow experiment is a deployment at weight zero receiving mirrored traffic. Stopping a deployment is bounding min and max replicas to 0/0.

Weight based traffic split The traffic split is a list of {deployment_id, weight} entries where a weight is per ready replica . The router computes each deployment's effective capacity as weight × ready_replicas and routes proportionally to this capacity. Walk through the diagram above: both deployments have weight 1 but deployment A has 1 ready replica (capacity 1), deployment B has 3 (capacity 3), so traffic follows a 25%/75% split . Equal weights mean equal per-replica load , not equal traffic share.

We designed it this way because it makes routing and scaling the same conversation: Autoscaling composes for free. When deployment A scales from 1 replica to 3, its capacity triples and it automatically absorbs proportionally more traffic. With naive percentages, a deployment pinned at "25%" would saturate when scaled down and be idle when scaled up. Per-replica load is what you control. Weight 1 vs weight 2 says "each replica of B should work twice as hard as each replica of A" Replicas that aren't ready don't count. A deployment mid-cold-start, or degraded to 0 ready replicas, contributes zero capacity, so traffic flows to what can actually serve.

Weights are positive numbers with no sum constraint so 0.7/0.3 and 700/300 describe identical routing. If you want fixed traffic shares regardless of replica count you can setup A/B experiment cohorts which allow for integer percents that sum to 100%. Setting a split is one PATCH with a field mask:

tg beta endpoints update $DEPLOYMENT_A --traffic-weight 1 tg beta endpoints update $DEPLOYMENT_B --traffic-weight 1

take a deployment out of rotation without scaling it down

tg beta endpoints update $DEPLOYMENT_B --traffic-weight 0

⚠️ Important

A brand-new deployment will not get any traffic until it appears in the endpoint 's traffic split. If you create an endpoint and a deployment and have it READY, send a request, and get back a routing_error it is probably because the traffic split is not specified and the router hasn’t yet been told that this deployment should receive traffic. The CLI' s together beta endpoints deploy sets the split for you, which is why the quickstart "just works" but the raw-API/SDK path needs this set . Under the hood: choosing a config You don't need to write configs from scratch. Every model in the supported-models catalog ships with deployment profiles which are certified model + config pairs we benchmark and continuously improve:

tg beta models public zai-org/GLM-5.2 --json | jq '.data[0].deploymentProfiles[]'

or, once you have the model ID, list its deployable configs directly:

tg beta models configs ml_CcEtnFUbitJYNja6TTR6U

{ "profileId": "profile-cabcbcec874c", "certifiedConfigRevisionId": "cr_Cd35DNpQuHM3RihtCkN59", "certifiedModelRevisionId": "rv_CcEuPu4Gk6c4M7v4PTYuG", "gpuType": "NVIDIA-B200", "gpuCount": 4, "quantization": "FP4", "performanceBenchmarks": {}, "config": "projects/proj_CbdXDuLRFhUPv1mGfxwUc/configs/cr_Cd35DNpQuHM3RihtCkN59", "model": "projects/proj_CbdXDuLRFhUPv1mGfxwUc/models/ml_CcEtnFUbitJYNja6TTR6U/revisions/rv_CcEuPu4Gk6c4M7v4PTYuG", "parallelism": "TP4" }

From there you can copy the model and config resource names into a deployment create:

tg beta endpoints deploy zai-org/GLM-5.2 \ --endpoint my-glm-endpoint \ --config cr_Cd35DNpQuHM3RihtCkN59 \ --traffic-weight 1

When you're choosing between configs, the selectors tell you what each recipe optimizes:

Selector Values The tradeoff it encodes

accelerator_type H100, H200, B200, … Cost vs memory bandwidth vs availability

accelerator_count 1, 2, 4, 8 Fit the model + KV-cache headroom

optimization latency / throughput / balanced The fundamental serving tradeoff (below)

topology e.g. aggregated/disaggregated How the model splits across GPUs

The optimization axis is the one...