TensorRT-LLM alternatives

Closest: Inference Optimization

• 

  llama.cpp

  Open-source LLM inference in C/C++, plus the Llama app for your desktop

  CuratedLinuxMacWindows
  1 use thisCompare with TensorRT-LLM
• 

  BentoML

  Framework for building ML-powered services

  Curated
  0 use thisCompare with TensorRT-LLM
• 

  Escha Labs

  2-bit compressed LLMs that run 27B to 35B models on one consumer GPU

  CuratedLinux
  0 use thisCompare with TensorRT-LLM
• 

  llm-d

  Kubernetes-native distributed LLM inference stack

  Curated
  0 use thisCompare with TensorRT-LLM
• 

  NVIDIA Dynamo

  Distributed inference serving framework for LLMs on GPU clusters

  Curated
  0 use thisCompare with TensorRT-LLM
• 

  NVIDIA Dynamo-Triton

  NVIDIA open-source inference serving software

  CuratedLinuxWindows
  0 use thisCompare with TensorRT-LLM
• 

  ONNX Runtime

  Cross-platform ML model accelerator

  CuratedLinuxMacWindows
  0 use thisCompare with TensorRT-LLM
• 

  Pruna AI

  Open-source framework that makes AI models faster and smaller, plus a video API

  CuratedLinuxMacWindows
  0 use thisCompare with TensorRT-LLM
• 

  Ray Serve

  Scalable model serving with Ray

  Curated
  0 use thisCompare with TensorRT-LLM
• 

  Seldon Core

  Kubernetes-native ML deployment platform

  Curated
  0 use thisCompare with TensorRT-LLM
• 

  SGLang

  Fast serving framework for LLMs and VLMs with structured generation support

  Curated
  0 use thisCompare with TensorRT-LLM
• 

  Sol Engine

  NVIDIA's training-free acceleration framework for video diffusion inference

  Curated
  0 use thisCompare with TensorRT-LLM
• 

  TensorRT

  NVIDIA's SDK for high-performance inference

  Curated
  0 use thisCompare with TensorRT-LLM
• 

  TorchServe

  PyTorch model serving framework (repository archived)

  Curated
  0 use thisCompare with TensorRT-LLM
• 

  vLLM

  High-throughput and memory-efficient LLM serving

  Curated
  0 use thisCompare with TensorRT-LLM

More in Inference & Compute

• 

  Ollama

  Run local LLMs (Llama, Qwen, Gemma, and more) with a single command

  CuratedLinuxMacWindows
  2 use thisCompare with TensorRT-LLM
• 

  Akash Network

  Decentralized cloud compute marketplace for AI workloads on GPU nodes

  Curated
  0 use thisCompare with TensorRT-LLM
• 

  AWS SageMaker

  Amazon's fully managed ML platform

  CuratedWeb
  0 use thisCompare with TensorRT-LLM
• 

  Azure ML

  Microsoft Azure's cloud ML service

  Curated
  0 use thisCompare with TensorRT-LLM
• 

  Baseten

  Inference platform for deploying and scaling AI models

  Curated
  0 use thisCompare with TensorRT-LLM
• 

  Cloudflare AI

  Serverless AI inference on Cloudflare's global network (Workers AI)

  CuratedWeb
  0 use thisCompare with TensorRT-LLM
• 

  CoreWeave

  GPU-accelerated cloud for AI/ML workloads

  CuratedWeb
  0 use thisCompare with TensorRT-LLM
• 

  Crusoe

  AI infrastructure and GPU cloud provider

  Curated
  0 use thisCompare with TensorRT-LLM
• 

  Databricks

  Unified data analytics and AI platform

  CuratedWeb
  0 use thisCompare with TensorRT-LLM

Questions