Computer Vision Explained: How AI Sees Images

In short: computer vision is the part of AI that gets information out of images and video - recognising what is in a photo, finding where each object is, outlining it pixel by pixel, or reading text from a screenshot. A computer sees an image as a grid of numbers, and a vision model is trained on many labelled images until it learns which patterns of numbers go with which things. Face unlock, photo search and the image input in chatbots such as ChatGPT, Claude and Gemini are all computer vision.

What is computer vision, in simple words?

Computer vision is teaching a computer to understand pictures. A person glances at a photo and knows it shows a dog on a beach. A computer only receives numbers; computer vision is the set of methods that turns those numbers into answers such as "dog", "beach", or "the dog is in the lower left".

It is the "seeing" side of AI, alongside the "reading and writing" side covered in how LLMs work. Most modern computer vision is machine learning: the model learns from examples rather than from rules a programmer writes.

How does a computer see an image?

To a computer, an image is a grid of numbers, one or three per pixel. A greyscale pixel is a single brightness value; a colour pixel usually has three values, for red, green and blue. A 1,000 × 1,000 colour photo is therefore three million numbers.

A vision model takes that grid and passes it through layers of calculations. Early layers respond to simple patterns such as edges and colour changes; later layers combine them into textures, shapes and finally whole objects. Nobody writes these patterns in by hand - the model works them out during training.

What are the main computer vision tasks?

The four core tasks, from simplest to hardest, are classification, detection, segmentation and video understanding.

Related tasks build on these: optical character recognition (OCR) reads printed or handwritten text from images; face recognition matches a face against known faces; pose estimation finds the positions of a person's joints.

How are computer vision models trained?

A vision model is shown a very large number of labelled images and adjusts itself each time it gets one wrong. The loop is the same one described in machine learning basics: guess, measure the error, nudge the model's internal numbers, repeat.

Labelled datasets made this possible. ImageNet, organised by the WordNet hierarchy of English nouns, lists 14,197,122 images in 21,841 categories. In 2012 a deep neural network known as AlexNet won the ImageNet recognition challenge by a wide margin, the result usually credited with starting the deep-learning era.

What kinds of model are used for computer vision?

Three families of model come up most often: convolutional neural networks, vision transformers and promptable segmentation models.

• Convolutional neural networks (CNNs) slide small filters across the image to detect local patterns, then stack many such layers. AlexNet was a CNN. U-Net (2015), a CNN designed for medical images, is still a standard layout for segmentation.
• Real-time detectors. YOLO ("You Only Look Once"), published in 2015, finds all the objects in an image in a single pass; its authors report the base model running at 45 frames per second.
• Vision transformers (ViT). The Vision Transformer paper (2020) cut an image into 16 × 16-pixel patches and fed them to a transformer, the same design behind language models. See Vision Transformers explained.
• Promptable segmentation. Meta's Segment Anything (2023) outlines whatever object you point at or box. Its authors built a dataset of over 1 billion masks on 11 million images to train it. See Segment Anything (SAM) explained.

Where is computer vision used?

• Phones - face unlock, portrait-mode background blur, searching your photos for "beach" or "dog".
• Cars - lane-keeping, reading speed-limit signs, detecting pedestrians.
• Health - helping read X-rays, scans and microscope slides.
• Shops and factories - spotting faulty parts on a production line, self-checkout cameras.
• Documents - scanning receipts, forms and handwriting into text (OCR).
• Farming and the environment - counting crops or animals from drone and satellite images.
• Accessibility - apps that describe a scene or read a sign aloud for blind and low-vision users.

Can ChatGPT, Claude and Gemini see images?

Yes - current chatbot models are multimodal, which means the same model can take images as well as text. You can share a photo, chart or screenshot and ask about it: describe it, read the text in it, explain a graph, or spot a bug in a screenshot of code. Each model's page in our AI models directory lists the input types its maker documents.

Image generation is the reverse direction - text in, picture out - and is done by models such as FLUX 3 and Nano Banana. Generation and understanding are separate abilities, though some models have both.

Where does computer vision still struggle?

Vision models are weakest on images unlike their training data, on fine distinctions, and on groups their data under-represented.

• Unfamiliar images. A model trained mostly on clear, well-lit photos can fail on odd angles, poor lighting, drawings or heavily edited pictures.
• Fine-grained differences. Telling apart similar dog breeds or near-identical manufactured parts needs data collected specifically for that distinction.
• Uneven accuracy across people. The 2018 Gender Shades study tested three commercial gender-classification systems and found error rates of up to 34.7% for darker-skinned women, against a maximum of 0.8% for lighter-skinned men. Our AI ethics guide covers bias in more depth.
• Confident mistakes. A model can label an image wrongly with high confidence, so outputs used for safety or medical decisions are normally checked by a person.

Words to know

• Pixel - one dot of an image, stored as one or three numbers.
• Classification - giving a whole image a label.
• Object detection - finding objects and drawing a box around each.
• Segmentation - labelling every pixel by the object it belongs to.
• Bounding box - the rectangle a detector draws around an object.
• OCR - optical character recognition; reading text from images.
• CNN - convolutional neural network, the classic vision model design.
• Vision transformer (ViT) - a transformer that reads an image as a sequence of patches.
• Multimodal - a model that accepts more than one kind of input, such as text and images.

Related guides

• What is AI? A beginner's guide
• Machine learning basics
• Vision Transformers (ViT) explained
• Segment Anything (SAM) explained