# Inference & Compute

> The infrastructure layer that runs AI: GPU cloud providers, inference optimization libraries, distributed training infrastructure, purpose-built chips, and local LLM runtimes like llama.cpp and LM Studio. Renting compute? Compare the GPU clouds. Serving models in production? Start with vLLM and the inference optimizers. Running LLMs locally or at the edge? The local inference section covers runtimes and on-device tools. If it makes models run faster or cheaper, it's here.

43 tools in 5 subcategories. Web page: https://ailandscape.org/category/inference-compute · Each tool also has a markdown version at /tool/{slug}.md

## Cloud & GPU Providers

- [Akash Network](https://ailandscape.org/tool/akash-network): Decentralized cloud compute marketplace for AI workloads on GPU nodes
- [AWS Bedrock](https://ailandscape.org/tool/aws-bedrock): AWS managed service for foundation models from Anthropic, Meta, and Mistral
- [AWS SageMaker](https://ailandscape.org/tool/aws-sagemaker): Amazon's fully managed ML platform
- [Azure ML](https://ailandscape.org/tool/azure-ml): Microsoft Azure's cloud ML service
- [BentoML](https://ailandscape.org/tool/bentoml): Framework for building ML-powered services
- [Cloudflare AI](https://ailandscape.org/tool/cloudflare-ai): AI inference at the edge via Cloudflare Workers AI with zero cold starts
- [CoreWeave](https://ailandscape.org/tool/coreweave): GPU-accelerated cloud for AI/ML workloads
- [Databricks](https://ailandscape.org/tool/databricks): Unified data analytics and AI platform
- [Fireworks AI](https://ailandscape.org/tool/fireworks-ai): Fast, affordable inference API for open-source models with sub-100ms latency
- [Google Vertex AI](https://ailandscape.org/tool/google-vertex-ai): Google Cloud's unified ML platform
- [Heurist](https://ailandscape.org/tool/heurist): Decentralized GPU network for low-latency, verifiable AI model inference
- [Nebius](https://ailandscape.org/tool/nebius): AI cloud with GPU clusters and Token Factory managed inference
- [Ray Serve](https://ailandscape.org/tool/ray-serve): Scalable model serving with Ray
- [RunPod](https://ailandscape.org/tool/runpod): Developer-focused GPU cloud for training, inference, and serverless AI
- [SambaNova](https://ailandscape.org/tool/sambanova): Full-stack AI platform with custom RDU chips for fast inference at scale
- [Seldon Core](https://ailandscape.org/tool/seldon-core): Kubernetes-native ML deployment platform
- [TorchServe](https://ailandscape.org/tool/torchserve): PyTorch model serving framework
- [Triton](https://ailandscape.org/tool/triton): NVIDIA's open-source inference serving software

## Inference Optimization

- [llama.cpp](https://ailandscape.org/tool/llama-cpp): LLM inference in C/C++ for local deployment
- [ONNX Runtime](https://ailandscape.org/tool/onnx-runtime): Cross-platform ML model accelerator
- [SGLang](https://ailandscape.org/tool/sglang): Fast serving framework for LLMs and VLMs with structured generation support
- [Sol Engine](https://ailandscape.org/tool/sol-engine): NVIDIA's training-free acceleration framework for video diffusion inference
- [TensorRT](https://ailandscape.org/tool/tensorrt): NVIDIA's SDK for high-performance inference
- [TensorRT-LLM](https://ailandscape.org/tool/tensorrt-llm): NVIDIA's LLM inference framework for NVIDIA GPUs with TensorRT acceleration
- [vLLM](https://ailandscape.org/tool/vllm): High-throughput and memory-efficient LLM serving

## Training Infrastructure

- [DeepSpeed](https://ailandscape.org/tool/deepspeed): Deep learning optimization library by Microsoft
- [FSDP](https://ailandscape.org/tool/fsdp): Fully Sharded Data Parallel training in PyTorch
- [Horovod](https://ailandscape.org/tool/horovod): Distributed deep learning training framework
- [Megatron-LM](https://ailandscape.org/tool/megatron-lm): NVIDIA's large-scale transformer training
- [Nanotron](https://ailandscape.org/tool/nanotron): Minimalist large language model training by Hugging Face

## Compute & Hardware

- [Apple MLX](https://ailandscape.org/tool/apple-mlx): Array framework for Apple Silicon ML research
- [Cerebras WSE](https://ailandscape.org/tool/cerebras-wse): Wafer-scale engine for fastest LLM training and inference
- [CUDA](https://ailandscape.org/tool/cuda): NVIDIA's parallel computing platform
- [Groq LPU](https://ailandscape.org/tool/groq-lpu): Language Processing Unit for ultra-fast LLM inference
- [ROCm](https://ailandscape.org/tool/rocm): AMD's open software platform for GPU computing
- [Tenstorrent](https://ailandscape.org/tool/tenstorrent): AI hardware with open-source software stack

## Local & Edge Inference

- [EXO](https://ailandscape.org/tool/exo): Run frontier AI on your own devices by clustering everyday hardware
- [FastChat](https://ailandscape.org/tool/fastchat): Distributed multi-model LLM serving system with OpenAI-compatible API and web UI
- [Harbor](https://ailandscape.org/tool/harbor): Effortlessly run LLM backends, APIs, frontends, and services with one command
- [LM Studio](https://ailandscape.org/tool/lm-studio): Desktop app for discovering, downloading, and running open-weight LLMs locally
- [ollama](https://ailandscape.org/tool/ollama): Run local LLMs (Llama, Qwen, Gemma, and more) with a single command
- [Open WebUI](https://ailandscape.org/tool/open-webui): Self-hosted chat UI for local and cloud LLMs; Ollama, OpenAI-compatible API, RAG
- [SillyTavern](https://ailandscape.org/tool/sillytavern): Locally installed UI for text, image, and voice LLMs with deep customization

---

Part of [AI Landscape](https://ailandscape.org), an open map of the AI ecosystem. Index for AI assistants: https://ailandscape.org/llms.txt