# Multimodal & Perception

> Models that see and hear: vision-language models like GPT-4o and LLaVA, computer vision systems like SAM and Detectron2, and speech recognition engines like Whisper and Deepgram. Pure perception lives here; generative image and audio model weights belong to Foundation Models.

22 tools in 3 subcategories. Web page: https://ailandscape.org/category/multimodal-perception · Each tool also has a markdown version at /tool/{slug}.md

## Vision-Language Models

- [Flamingo](https://ailandscape.org/tool/flamingo): DeepMind's visual language model
- [Gemini Pro Vision](https://ailandscape.org/tool/gemini-pro-vision): Google's multimodal model with vision capabilities
- [GPT-4o](https://ailandscape.org/tool/gpt-4o): OpenAI's omni model with audio, vision, and text capabilities
- [GPT-4V](https://ailandscape.org/tool/gpt-4v): OpenAI's vision-capable language model
- [InternVL 2](https://ailandscape.org/tool/internvl-2): Strong open-source vision-language model by Shanghai AI Lab
- [LLaVA](https://ailandscape.org/tool/llava): Open-source visual instruction tuning model
- [Qwen-VL](https://ailandscape.org/tool/qwen-vl): Alibaba's visual language model with strong document understanding
- [Reka Core](https://ailandscape.org/tool/reka-core): Reka's multimodal model handling text, images, video, and audio

## Computer Vision

- [Detectron2](https://ailandscape.org/tool/detectron2): Meta's object detection and segmentation framework
- [DETR](https://ailandscape.org/tool/detr): End-to-end object detection with transformers by Meta
- [Google Vision AI](https://ailandscape.org/tool/google-vision-ai): Google Cloud's image recognition and OCR API
- [PaddleOCR](https://ailandscape.org/tool/paddleocr): Multilingual OCR toolkits based on PaddlePaddle
- [SAM](https://ailandscape.org/tool/sam): Segment Anything Model by Meta
- [SAM 2](https://ailandscape.org/tool/sam-2): Meta's model for image and video segmentation
- [Tesseract](https://ailandscape.org/tool/tesseract): Open-source OCR engine
- [YOLO](https://ailandscape.org/tool/yolo): Real-time object detection by Ultralytics

## Speech Recognition

- [AssemblyAI](https://ailandscape.org/tool/assemblyai): Speech AI models for transcription and understanding
- [Azure Speech](https://ailandscape.org/tool/azure-speech): Microsoft Azure's speech-to-text and translation service
- [Deepgram](https://ailandscape.org/tool/deepgram): AI-powered speech recognition API
- [Parakeet](https://ailandscape.org/tool/parakeet): NVIDIA's NeMo ASR model with state-of-the-art accuracy
- [superwhisper](https://ailandscape.org/tool/superwhisper): Offline AI voice-to-text dictation for macOS, Windows, and iOS
- [Whisper](https://ailandscape.org/tool/whisper): OpenAI's open-source speech recognition model

---

Part of [AI Landscape](https://ailandscape.org), an open map of the AI ecosystem. Index for AI assistants: https://ailandscape.org/llms.txt