Novita AI

Thesis

Novita AI is executing a two-pronged evolution: it operates a commercial model API and agent sandbox platform for third-party frontier models, while simultaneously building deep inference infrastructure—most visibly pegaflow, a Rust-based KV cache storage engine with vLLM integration—that targets the performance bottleneck of large-scale LLM serving. The pattern of GTM hiring in San Mateo alongside a relentless pegaflow release cadence and a fork portfolio concentrated on inference kernels (sglang, flash-attention, FlashMLA, llm-d) suggests Novita is commoditizing model access at the API layer while differentiating on inference systems engineering underneath. The thin evidence on research hires and the archival of earlier generative-media SDKs (Python, JavaScript, Go) further supports a strategic narrowing toward infrastructure and platform, away from the broad creative-AI toolkit the company shipped in 2023–2024.

Signal desks

Hiring

• GTM & Sales buildout in San Mateo: Four open roles—Forward Deployed Engineer (Sales team) P1E1, Solutions Engineer (AI Cloud Infrastructure, GTM team) P5E60, Account Executive (GTM team, hybrid) P6E46, and Chief of Staff (Operations) P7E39. All on-site or hybrid in San Mateo; none remote-first. The FDE role explicitly mentions Model APIs, GPU infrastructure, and Agent Sandbox environments as deployment surface P1. The Account Executive JD references "Sales-Led Growth (SLG) motion" targeting AI startups and mid-market enterprises, with cross-border collaboration between US clients and a global engineering team "based in China and the US" P6.
• No cited engineering or research roles in this pack. The absence of open engineering, ML research, or infrastructure platform roles alongside active GTM hiring implies the core technical team may be established (or located outside the US), and the current priority is commercialization.

Forks

• Inference engine & kernel forks: sgl-project/sglang E53, Dao-AILab/flash-attention E55, deepseek-ai/FlashMLA E56, and sgl-project/SpecForge E57—all forked on 2025-10-27—signal deep work on attention kernels and inference serving. The llm-d/llm-d and llm-d/llm-d-router forks E58E59 point to disaggregated inference scheduling exploration. The ai-dynamo/dynamo fork E40 suggests NVIDIA Dynamo investigation.
• Vendor verifier forks: MoonshotAI/Kimi-Vendor-Verifier E36, MoonshotAI/K2-Vendor-Verifier E35, and MiniMax-AI/MiniMax-Provider-Verifier E34—all forked April 2026—indicate Novita is validating third-party model provider APIs it onboards onto its platform.
• Infrastructure & training forks: kubernetes-sigs/gateway-api-inference-extension E25 suggests work on Kubernetes-native inference routing. KellerJordan/modded-nanogpt E10 is a training-research fork, isolated and low-profile.

Releases

• pegaflow dominates: At least 20 releases between early April and late June 2026, from v0.0.18 through v0.22.10 [E4–E8, E11, E13, E22–E24, E26, E28, E32–E33, E38, E41–E42, E44]. The v0.22.4 release (May 29, 2026) introduced disaggregated prefill/decode over RDMA push with claimed 2–4× TTFT improvement vs. NIXL P27, plus query leases replacing pin/unpin semantics and save-only mode. v0.22.10 (June 22, 2026) added MLA KV page-first storage, per-layer MLA TP save distribution, model-aware KV transfer backend selection, and metaserver block-redundancy observability metrics P3. The release notes consistently surface production-grade concerns: strict version handshakes, NUMA-aware memory, Prometheus metrics, and cross-node replication visibility.
• sglang fork releases: Three tagged releases—0.4.1 ("For vllm image") P21E37, 0.4.2 P25E29, and 0.4.2.post2 P23E20—all described as automated wheel uploads from novitalabs/vllm-int, suggesting Novita maintains a patched sglang build for its vLLM integration pipeline.
• novita-cli v0.1.0: Initial PyPI release (April 29, 2026) providing a unified CLI covering text, image, video, audio, GPU sandbox runtimes, serverless endpoints, templates, storage, account, and billing P22E30.
• dify-plugin-novita v0.0.6: Release for Dify platform integration, no published notes P20.

Talking

• Third-party model onboarding narrative: Blog posts in June 2026 position Novita as the API access layer for GLM 4.6V (vision + tool calling) W2, GLM 5.2 (1M-token context, coding focus) W3, MiniMax M3 (1M-context, agentic AI) W1, and Nemotron 3 Nano 30B A3B (256K context, function calling, reasoning) W4. Every post emphasizes OpenAI-compatible chat completions and lists model IDs for drop-in usage.
• Agent sandbox positioning: A May 2026 post walks through running Harbor agent evaluations on Novita Agent Sandbox, describing a cloud sandbox runtime for AI agents that execute generated code with multi-language execution, pause/resume, and SDK/CLI management W6.
• Startup program: A dedicated startup credits program offers up to $10,000 split across Model APIs and Agent Sandbox, with upfront and matched credits W5.
• No cited non-blog public discourse: No HN discussion, conference talks, academic papers, or social-media traction cited in this pack.

Shipping

• pegaflow is Novita's most substantive shipped artifact: A Rust-based, Apache-2.0 licensed KV cache storage engine for LLM inference with GPU offloading, SSD caching, and cross-node RDMA sharing, working as a drop-in connector for vLLM P28E47. A joint blog post with the vLLM team (May 18, 2026) provides external validation P28. The engine ships as a Python package (pegaflow-llm) with CUDA 12 and 13 variants P28.
• novita-cli: Python CLI unifying management of model APIs, GPU instances, sandbox runtimes, serverless endpoints, and billing P22.
• Model API platform: The platform serves serverless inference for models from Z.ai/GLM, MiniMax, NVIDIA, and others with OpenAI-compatible endpoints, function calling, structured outputs, and reasoning support W1W2W3W4.
• Agent Sandbox: A cloud sandbox runtime for AI agent code execution, positioned for eval workflows like Harbor W6, with MCP server integration for GPU instance management P16.
• Earlier SDKs archived: The Python SDK P10, JavaScript SDK P12, and Golang SDK P9 are all archived, signaling deprecation of the original generative-media API surface (Txt2Img, Img2Img, ControlNet, etc.) in favor of the current model API + infrastructure platform.

Research themes

Evidence for active research is thin; the strongest signals come from the fork portfolio and pegaflow release notes:

• Disaggregated inference: The pegaflow v0.22.4 release introduces disaggregated prefill/decode with RDMA push, layer-by-layer KV transfer overlapping with compute P27. The llm-d/llm-d and llm-d/llm-d-router forks E58E59 suggest continued exploration of disaggregated serving architectures.
• MLA (Multi-head Latent Attention) optimization: pegaflow v0.22.9 and v0.22.10 include MTP split connector support, MLA KV page-first storage, and per-layer MLA TP save distribution P3P4, paired with the deepseek-ai/FlashMLA fork E56. This points to production support for DeepSeek-style attention architectures.
• Inference autotuning: The autotuner repo (MIT license, actively maintained with last push June 10, 2026) targets SGLang and vLLM parameter optimization, claiming 60%+ throughput gains versus defaults P18E51.
• KV cache storage systems: pegaflow's architecture—decoupled sidecar, topology-aware NUMA transfers, GIL-free Rust core, lease-backed query semantics, Prometheus/OTLP observability—represents a systems research contribution targeting the KV cache bottleneck P28P3P27.
• No cited model training or pretraining research: The modded-nanogpt fork E10 is the only training-adjacent signal and is isolated. No papers, model cards, or training infrastructure releases are cited.

Hiring & scaling

• San Mateo as North American commercial hub: All four open roles are in San Mateo, with the Account Executive role explicitly hybrid/remote-capable P6. The Chief of Staff JD references "the North American team" and "local compliance" P7, confirming a US entity buildout. The Account Executive role mentions cross-border collaboration with engineering teams "based in China and the US" P6.
• Sales-Led Growth motion: The Account Executive role describes a "full-cycle" closer hunting AI startups and mid-market enterprises for GPU compute and LLM API solutions P6. This is a dedicated outbound enterprise sales motion, not self-serve PLG.
• Customer-facing technical roles dominate: Forward Deployed Engineer ("work at the intersection of engineering, customer success, and product") P1 and Solutions Engineer ("primary technical leader and trusted advisor for customers") P5 both emphasize pre-sales, POCs, integration, and customer feedback loops.
• No engineering hiring cited: The absence of software engineering, infrastructure, or research roles in this pack is notable given the depth of pegaflow development. The evidence gap suggests either a separate hiring pipeline (potentially China-based) or a stable existing team not captured in this evidence set.
• Operations hire signals organizational maturity: The Chief of Staff role (compliance, budget tracking, SaaS management, contract review) P7 suggests Novita is building the operational scaffolding for a scaling US entity.

Category implications

• Neocloud/inference infrastructure: pegaflow is a direct competitive signal in the KV cache storage and disaggregated inference category. Its vLLM integration and joint blog post with the vLLM team P28 position it as an alternative to NVIDIA NIXL and in-engine cache solutions. The claimed 2–4× TTFT improvement over NIXL on H20/Qwen3-8B P27 is an aggressive performance claim for an independent neocloud. The rapid release cadence (20+ releases in ~3 months) suggests a dedicated, well-resourced systems team [E4–E8, E11, E13, E22–E24, E26, E28, E32–E33, E38, E41–E42, E44].
• Model API platform: Novita's model onboarding strategy—GLM 4.6V, GLM 5.2, MiniMax M3, Nemotron 3 Nano —positions it as a multi-model serverless API provider competing with Together AI, Fireworks, and DeepInfra. The emphasis on OpenAI-compatible endpoints with function calling, structured outputs, and reasoning W1W2W4 targets drop-in developer adoption. The vendor verifier forks suggest Novita validates upstream model providers before listing them.
• Agent infrastructure: The Agent Sandbox product W6 and MCP server for GPU instance management P16 target the growing agent evaluation and execution market. The startup program's equal credit split between Model APIs and Agent Sandbox W5 indicates these are co-equal platform pillars.
• No evidence of proprietary model training: All blog posts highlight third-party models. The archival of generative-media SDKs P9P10P12 suggests Novita has exited the creative-AI API business in favor of infrastructure and model serving. This distinguishes it from labs that train frontier models alongside serving infrastructure.
• China-US cross-border structure: The Account Executive JD's reference to "global product, engineering, and marketing teams (based in China and the US)" P6 and the Chief of Staff's Mandarin fluency requirement P7 confirm a cross-border organizational structure, relevant for export control, data residency, and geopolitical risk assessment.

Traction highlights

• pegaflow: 136 stars, 20 forks, 36 open issues on GitHub P28. Joint blog post with the vLLM team (May 2026) P28. Active community engagement evidenced by issue volume and rapid release iteration.
• AnimateAnyone: 779 stars, 69 forks—Novita's most-starred public repo, an unofficial implementation of Animate Anyone P13E43.
• sd-webui-cleaner: 344 stars, 26 forks—a Stable Diffusion WebUI extension P11E45.
• autotuner: 11 stars, 3 forks, 4 open issues, active development (last push June 10, 2026) P18E51.
• novita-mcp-server: 11 stars, 9 forks, Smithery badge P16E52.
• Startup program: Up to $10,000 in credits with staged matching W5, signaling a GTM motion to acquire developer teams.
• Model API breadth: At least four third-party model families live on the platform as of June 2026 (GLM, MiniMax, NVIDIA Nemotron, Z.ai) .
• Evidence gap: No revenue, customer count, inference volume, or valuation data cited. No Hugging Face model download, PyPI download, or Docker pull metrics beyond GitHub stars.