Image Data Representation: Pixels, Features, and Structure

When we look at a photograph or a frame from a video, our brains effortlessly process the scene, identifying objects, colors, and their relationships. For an AI system to do something similar, it first needs a way to "see" and interpret this visual information. This involves converting images into a language machines understand: numbers. As the chapter introduction mentioned, one way to think about an image is as a grid of pixel values, where each pixel at coordinates $(x, y)$ has an intensity $I(x,y)$. Let's explore this and other ways images are represented for AI.

Pixels: The Raw Material of Images

At its most fundamental level, a digital image is a collection of pixels, short for "picture elements." Each pixel is the smallest unit of an image and holds information about the color or intensity at a specific point.

Grayscale Images

The simplest type of image is a grayscale image, often called a black-and-white image. In a grayscale image, each pixel is represented by a single number. This number typically indicates the intensity of light, ranging from 0 (black) to 255 (white), with various shades of gray in between. So, for an image with width $W$ and height $H$, we can imagine it as a 2D array or matrix of numbers. If $I(x,y)$ is the intensity of the pixel at row $y$ and column $x$:

A tiny 3x3 grayscale image might look like this if we wrote down its pixel values:

[ [ 10,  30,  50 ],
  [120, 150, 180 ],
  [200, 220, 255 ] ]

Here, the pixel at $(0,0)$ (top-left) is very dark (10), while the pixel at $(2,2)$ (bottom-right) is pure white (255).

Color Images

Color images are a bit more complex. They typically use a combination of primary colors to represent a full spectrum of hues. The most common model is RGB, which stands for Red, Green, and Blue. In the RGB model, each pixel has three values, one for the intensity of red, one for green, and one for blue. Each of these values also often ranges from 0 to 255.

So, a color pixel $P(x,y)$ can be represented as a set of three values: $(R(x,y), G(x,y), B(x,y))$.

• $(0, 0, 0)$ would be black (no red, no green, no blue).
• $(255, 255, 255)$ would be white.
• $(255, 0, 0)$ would be pure red.
• $(128, 128, 128)$ would be a medium gray.

A tiny 1x2 color image (1 pixel high, 2 pixels wide) might have values like:

[ [(255,0,0), (0,255,0)] ]  // First pixel is Red, Second pixel is Green

You can think of a color image as three separate grayscale images (one for each color channel) stacked on top of each other.

Image Dimensions

The dimensions of an image are usually described as width x height x channels.

• Width: The number of pixels horizontally.
• Height: The number of pixels vertically.
• Channels: The number of values per pixel. For a grayscale image, this is 1. For an RGB color image, this is 3.

So, a small color photo of 640 pixels wide, 480 pixels high would be represented by $640 \times 480 \times 3 = 921,600$ numbers! This can be a lot of data for an AI model to process directly.

Beyond Pixels: The Need for Features

While raw pixel values are the basic representation, they are not always the most effective input for AI models, especially for complex tasks. Here's why:

• High Dimensionality: As seen above, even small images consist of many numbers. This can make models slow to train and require a lot of data.
• Sensitivity: Small changes like a slight shift in lighting, or moving an object a few pixels, can drastically change many pixel values, even if the image content is essentially the same.
• Lack of Abstraction: Pixels tell us about color/intensity at a point, but they don't directly tell us about "objects," "textures," or "shapes."

To address this, we often extract features from images. Features are derived, more compact, and hopefully more informative representations of the image content.

Example: Color Histograms

A color histogram is a simple yet useful feature. It counts how many pixels in an image fall into certain color ranges or bins. For example, it can tell us the distribution of reds, greens, and blues in an image, or the overall brightness distribution.

A histogram doesn't care where the colors are in the image, just how much of each color (or intensity) is present. This makes it robust to changes like object rotation or translation.

A simplified color histogram showing the proportion of dominant red, green, and blue tones in an example image. This gives a general sense of the image's color palette.

Other Simple Features (Briefly)

Beyond color histograms, other types of features can be extracted:

• Edges: Edges are locations in an image where there's a sharp change in intensity or color. They often correspond to the boundaries of objects and are important visual cues. Algorithms can detect these edges and represent them.
• Texture: Texture refers to the visual patterns in an image, like the roughness of tree bark or the smoothness of silk. Features can be designed to quantify these patterns.
• Corners and Keypoints: Specific points in an image that are "interesting" or stable across different views of an object can be identified as features.

The goal of these engineered features is often to reduce the amount of data while retaining or even emphasizing the important information for a given task.

The Importance of Structure

Pixels are arranged in a grid, and this spatial arrangement, or structure, is fundamental. The relationships between neighboring pixels, and groups of pixels, define shapes, objects, and the overall scene.

• A collection of blue pixels at the top of an image might represent the sky.
• A circular arrangement of edge pixels might indicate a ball.

While features like color histograms discard spatial information, many AI techniques, particularly modern deep learning models like Convolutional Neural Networks (CNNs), are specifically designed to learn from and exploit this local spatial structure directly from the pixel data or from learned features. Understanding that images have inherent structure is important before diving into how models use this structure.

Image Embeddings: Capturing Semantics Numerically

In more advanced AI, especially with deep learning, models can learn to transform an entire image (or parts of it) into a dense list of numbers called an image embedding or a feature vector. This embedding is a relatively low-dimensional representation compared to the raw pixels, but it aims to capture the high-level semantic content of the image.

Imagine you have a 1000-pixel by 1000-pixel color image. That's 3 million raw pixel values! An image embedding might represent this same image using just a few hundred numbers. The key idea is that images with similar content (e.g., two different photos of cats) will have embeddings that are "close" to each other in this numerical space, while images with different content (e.g., a cat and a car) will have embeddings that are "far apart."

We won't go into how these embeddings are learned in this introductory course, but it's useful to know they exist. They are a powerful way to represent images for tasks like image search, classification, and, importantly for multimodal AI, for comparing and combining image information with other data types like text.

In summary, representing images for AI starts with the raw pixel data. From there, we can extract various features to get more abstract information, always keeping in mind the inherent structure of the image. And for more sophisticated understanding, AI models can learn compact, meaning-rich embeddings. These different representations provide the foundation for AI to "understand" and work with visual information.