Skip to content
← All questions
AI-generated draftEvidence through 2026-11-10Uncertainty: highRe-checked automatically every 2 weeks

Answer

First, a definitional fork

"Inference engine" means two different things, and the state of the art differs sharply:

  • AI/LLM serving engines — the runtime stack that executes model forward passes (this is almost certainly what's being asked).
  • Logic/rule engines — Datalog/Prolog-style deductive engines.

LLM serving: the field has converged, then re-differentiated

By late 2026 the "big three" open engines — vLLM, SGLang, TensorRT-LLM — have largely converged on the same core feature set: continuous/in-flight batching, paged KV cache, prefix caching, chunked prefill, tensor/pipeline/data/expert parallelism, FP8/FP4 quantization, speculative decoding, and prefill–decode (PD) disaggregation. Differentiation is now mostly about architectural bets and operational tradeoffs:

| Engine | Bet | Where it wins | |---|---|---| | vLLM (PagedAttention) | Breadth and iteration speed | 400+ architectures, day-one model support, NVIDIA/AMD/TPU/Gaudi/CPU, sub-minute cold start. The default answer for most teams. | | SGLang (RadixAttention) | Cross-request prefix reuse + runtime | High-concurrency MoE (DeepSeek/Qwen scale), best TTFT with shared prefixes, scheduler-level constrained/JSON decoding, tightly integrated expert parallelism on NVL72-class systems. | | TensorRT-LLM | Ahead-of-time compilation | Highest raw NVIDIA throughput (typically ~15–25% over vLLM on Hopper for dense models), lowest TTFT at high concurrency — at the cost of lengthy engine builds, narrow model coverage, NVIDIA-only. Its PyTorch backend (default since v1.0) trades some peak throughput for much easier deployment. |

Edge and secondary players: llama.cpp/Ollama/MLX for local and Apple-silicon inference; LMDeploy and TGI (now largely in maintenance-style decline relative to the leaders); newer latency-focused entrants such as TokenSpeed, which claims an edge over TensorRT-LLM on B200 for agentic coding workloads.

The real frontier is now the orchestration layer, not the single engine

The interesting work has moved up a level, to distributed serving controllers: NVIDIA Dynamo, llm-d (v0.5 in late 2026), and the Kubernetes Gateway API inference extension. These handle:

  • PD disaggregation at scale — separate prefill and decode pools, now the standard architecture; llm-d demonstrates wide expert-parallel decode across 16×16 prefill/decode topologies on B200.
  • KV-cache-aware / prefix-aware routing — send requests to the replica already holding their prefix, which matters more than raw kernel speed for RAG and multi-turn agents.
  • KV cache offload and tiering (GPU → CPU → storage), since decode is memory-bandwidth- and capacity-bound rather than compute-bound.

Accelerator techniques: what actually pays off

  • Speculative decoding is now standard (EAGLE-3, Medusa-style heads, n-gram/suffix decoding, trained draft models, adaptive draft lengths). Note a 2026 MLSys study arguing the production gains are often smaller than papers claim — acceptance rate collapses under distribution shift and at high batch sizes. Adaptive, self-tuning drafts (e.g., TIDE, SpecForge-trained drafts) are the response.
  • Quantization to FP8/FP4 with Blackwell-era formats, plus MoE-aware kernels.
  • Structured/constrained decoding integrated into the scheduler rather than bolted on as a token filter.

Honest caveats

Almost all public head-to-head numbers are vendor- or blog-sourced, run on a single H100/B200 with one model, and are highly sensitive to concurrency, prompt-length distribution, and batch shape. There is no universal winner; the practical advice in 2026 is: benchmark on your traffic, start with vLLM, trial SGLang if prefixes or structured output dominate, and consider TensorRT-LLM only with stable high-volume NVIDIA deployments and spare engineering capacity.

If you meant logic engines

That world is quieter but active: Soufflé remains the reference Datalog-to-native-C++ compiler for static analysis; FlowLog (2026) compiles Soufflé-style programs into Differential Dataflow for incremental maintenance; CozoDB packages Datalog as a transactional relational-graph-vector database. The trend there is incremental/streaming evaluation rather than raw one-shot speed.

Evidence

Sources (12)

Last checked 2026-11-10Source details

These sources were captured during web research; source type and confidence were not independently assessed.

  1. Captured during research
  2. Captured during research
  3. Captured during research
  4. Captured during research
  5. Captured during research
  6. SGLang

    sgl-project.github.io

    Captured during research
  7. TensorRT-LLM

    docs.nvidia.com

    Captured during research
  8. Captured during research
  9. Captured during research
  10. Captured during research
  11. Captured during research
  12. Captured during research
Living answer

Update history

  1. Initial answerVersion 1
    12 sources · AI-generated

Sign in to join the discussion

No comments yet on this answer. Comments and your saved library are available to signed-in readers.