Answer
First, a definitional fork
"Inference engine" means two different things, and the state of the art differs sharply:
- AI/LLM serving engines — the runtime stack that executes model forward passes (this is almost certainly what's being asked).
- Logic/rule engines — Datalog/Prolog-style deductive engines.
LLM serving: the field has converged, then re-differentiated
By late 2026 the "big three" open engines — vLLM, SGLang, TensorRT-LLM — have largely converged on the same core feature set: continuous/in-flight batching, paged KV cache, prefix caching, chunked prefill, tensor/pipeline/data/expert parallelism, FP8/FP4 quantization, speculative decoding, and prefill–decode (PD) disaggregation. Differentiation is now mostly about architectural bets and operational tradeoffs:
| Engine | Bet | Where it wins | |---|---|---| | vLLM (PagedAttention) | Breadth and iteration speed | 400+ architectures, day-one model support, NVIDIA/AMD/TPU/Gaudi/CPU, sub-minute cold start. The default answer for most teams. | | SGLang (RadixAttention) | Cross-request prefix reuse + runtime | High-concurrency MoE (DeepSeek/Qwen scale), best TTFT with shared prefixes, scheduler-level constrained/JSON decoding, tightly integrated expert parallelism on NVL72-class systems. | | TensorRT-LLM | Ahead-of-time compilation | Highest raw NVIDIA throughput (typically ~15–25% over vLLM on Hopper for dense models), lowest TTFT at high concurrency — at the cost of lengthy engine builds, narrow model coverage, NVIDIA-only. Its PyTorch backend (default since v1.0) trades some peak throughput for much easier deployment. |
Edge and secondary players: llama.cpp/Ollama/MLX for local and Apple-silicon inference; LMDeploy and TGI (now largely in maintenance-style decline relative to the leaders); newer latency-focused entrants such as TokenSpeed, which claims an edge over TensorRT-LLM on B200 for agentic coding workloads.
The real frontier is now the orchestration layer, not the single engine
The interesting work has moved up a level, to distributed serving controllers: NVIDIA Dynamo, llm-d (v0.5 in late 2026), and the Kubernetes Gateway API inference extension. These handle:
- PD disaggregation at scale — separate prefill and decode pools, now the standard architecture; llm-d demonstrates wide expert-parallel decode across 16×16 prefill/decode topologies on B200.
- KV-cache-aware / prefix-aware routing — send requests to the replica already holding their prefix, which matters more than raw kernel speed for RAG and multi-turn agents.
- KV cache offload and tiering (GPU → CPU → storage), since decode is memory-bandwidth- and capacity-bound rather than compute-bound.
Accelerator techniques: what actually pays off
- Speculative decoding is now standard (EAGLE-3, Medusa-style heads, n-gram/suffix decoding, trained draft models, adaptive draft lengths). Note a 2026 MLSys study arguing the production gains are often smaller than papers claim — acceptance rate collapses under distribution shift and at high batch sizes. Adaptive, self-tuning drafts (e.g., TIDE, SpecForge-trained drafts) are the response.
- Quantization to FP8/FP4 with Blackwell-era formats, plus MoE-aware kernels.
- Structured/constrained decoding integrated into the scheduler rather than bolted on as a token filter.
Honest caveats
Almost all public head-to-head numbers are vendor- or blog-sourced, run on a single H100/B200 with one model, and are highly sensitive to concurrency, prompt-length distribution, and batch shape. There is no universal winner; the practical advice in 2026 is: benchmark on your traffic, start with vLLM, trial SGLang if prefixes or structured output dominate, and consider TensorRT-LLM only with stable high-volume NVIDIA deployments and spare engineering capacity.
If you meant logic engines
That world is quieter but active: Soufflé remains the reference Datalog-to-native-C++ compiler for static analysis; FlowLog (2026) compiles Soufflé-style programs into Differential Dataflow for incremental maintenance; CozoDB packages Datalog as a transactional relational-graph-vector database. The trend there is incremental/streaming evaluation rather than raw one-shot speed.
EvidenceSources (12)
Last checked 2026-11-10Source details
Sources (12)
These sources were captured during web research; source type and confidence were not independently assessed.
- Captured during research
- SGLang vs vLLM vs TensorRT-LLM
www.truefoundry.com
Captured during research - vLLM vs TensorRT-LLM vs SGLang: Which Inference ...
www.runpod.io
Captured during research - vLLM vs SGLang vs TensorRT-LLM - Inference Engineering
inferenceengineering.tech
Captured during research - vLLM vs Ollama vs SGLang vs TensorRT-LLM - The AI Engineer
theaiengineer.substack.com
Captured during research - SGLang
sgl-project.github.io
Captured during research - TensorRT-LLM
docs.nvidia.com
Captured during research - Captured during research
- vLLM vs TensorRT-LLM vs SGLang: Which Is Fastest? ...
www.spheron.network
Captured during research - Soufflé vs Differential Datalog vs CozoDB in 2026: Which ...
www.pistack.xyz
Captured during research - Captured during research
- Captured during research
Update history
- Initial answerVersion 112 sources · AI-generated