Computer Vision Explained: How AI Sees the World

Computer vision is the field of AI concerned with getting machines to extract useful information from images and video - the "seeing" half of AI, as opposed to the "reading and writing" half covered in language models.

The basic building block: images as numbers

A digital image is just a grid of numbers - each pixel has a value (or three, for red/green/blue) representing its color. A computer vision model doesn't "see" a picture the way a person does; it processes that grid of numbers and learns to associate patterns in it with labels or outputs.

Core tasks in computer vision

• Image classification - assigning a label to an entire image ("this photo contains a dog"). This is the simplest core task and the one most beginner tutorials start with.
• Object detection - finding and localizing multiple objects within an image, typically drawing a bounding box around each one and labeling it ("dog at these coordinates, person at these coordinates"). This is a strictly harder problem than classification, because the model has to find objects, not just recognize that one is present somewhere.
• Image segmentation - going a level further than a bounding box, and labeling every individual pixel in the image according to what object it belongs to. Useful anywhere a precise outline matters, e.g. removing a background or measuring an object's exact shape.
• Video understanding - extending these tasks across time, tracking objects and recognizing actions across a sequence of frames rather than a single image.

How modern computer vision models are trained

Like most modern AI, current computer vision systems are trained with deep learning: a neural network is shown a large number of labeled images and adjusts itself to improve its predictions, using the same underlying mathematical principles as the training process described in Machine Learning Basics. Modern architectures increasingly borrow the same transformer-based attention mechanisms used in language models, applied to image data - a convergence worth knowing about if you're comparing vision and language model architectures directly.

Where computer vision shows up in current AI tools

Several current large language models are multimodal - Claude Opus 5 and GPT-5.6 both document image/vision input in their official specs (see our AI Models pages for the sourced details) - meaning the same model handling your text conversation can also interpret an image you share with it: reading a chart, describing a photo, or extracting text from a screenshot. This is functionally computer vision, integrated into a general-purpose assistant rather than a dedicated vision-only tool.

Where computer vision still struggles

• Unusual or out-of-distribution images. A model trained mostly on well-lit, centered photos can fail unpredictably on an image that looks nothing like its training data - unusual angles, poor lighting, artistic or heavily edited images.
• Fine-grained distinctions. Telling apart visually similar categories (specific dog breeds, similar-looking manufactured parts) is harder than broad classification and typically needs data specifically curated for that distinction.
• Demographic accuracy gaps. As noted in our AI Ethics guide, computer vision accuracy has been documented to vary across demographic groups when training data isn't representative - a real, tested failure mode worth checking for before deploying any vision system on people.