Data Representation
Introduction
Computers represent data using bits, commonly written as 0 and 1. By organizing and interpreting these bits according to agreed rules, we can represent text, images, video, and audio.
This article follows those rules from basic values to complete representations, beginning with images.
Text
Image
1. RGB and Color Representation
1.1 Light and Human Color Vision
Light possesses various spectral compositions. Upon entering the eye, light stimulates photoreceptor cells, and the visual system generates the perception of color based on these stimuli.
In humans with normal color vision, photopic (daylight) vision relies primarily on three types of cone cells, each with a distinct response to different ranges of light wavelengths. Under specified viewing conditions, the stimulation of these three types of cells by a given light source can be characterized by three quantities.
This implies that two light sources with different spectral compositions can still be perceived as the same color, provided they elicit the same responses in the three cone types under the same viewing conditions.
This principle provides a basis for display devices to reproduce a wide range of colors using a limited number of light sources.
1.2 RGB and Color Gamut
By selecting red, green, and blue as primary lights and independently controlling their intensities, display devices can generate a range of color perceptions.
For instance, an appropriate mixture of red and green light is perceived as yellow.
A color reproducible using this set of primary lights can be described by three components:
- Red Component
- Green Component
- Blue Component
The range of colors that a device can reproduce is its Color Gamut. A fixed set of three real primary lights cannot reproduce every color visible to the human eye through non-negative mixtures.
1.3 Channels and Bit Depth
The R, G, and B components are stored as three Channels. To store their values using finite space, we choose a Bit Depth for each Channel.
In RGB8, each Channel uses 8 bits, giving 256 encoded values from 0 to 255:
1 | R: 0 ... 255 |
A color can then be represented as:
1 | (255, 0, 0) |
In a specified RGB Color Space using this 8-bit representation, 255 is the maximum encoded value for a Channel, and 0 is the minimum. The range 0 to 255 is a choice of representation, rather than a universal requirement for all images.
1.4 From Color Values to Bits
Eight bits can represent an unsigned integer from 0 to 255. For example:
1 | R G B |
Therefore, one RGB8 color uses 24 bits, or 3 bytes.
2. Raster Image
2.1 Pixels and the 2D Grid
A Raster Image is a two-dimensional grid of Pixels. Each grid position has a color.
To describe the image, we need to describe both the positions of its Pixels and the colors at those positions.
For example, six Pixels named A through F can be arranged in three columns and two rows:
1 | A B C |
Each letter represents a Pixel with its own RGB value. Changing the arrangement can change the image, even if the set of colors stays the same.
2.2 Width, Height, and Coordinates
The number of columns is the Width, and the number of rows is the Height.
1 | x -> |
Common conventions used here are:
- The upper-left corner is the Origin, with Coordinates
(0, 0). - x increases to the right.
- y increases downward.
- x ranges from 0 to
Width - 1. - y ranges from 0 to
Height - 1.
Pixel E has Coordinates (1, 1): the second column and
the second row.
1 | Pixel Count = Width * Height |
Conceptually, each Pixel has a position and a color:
1 | Position: (x, y) |
In the layout used below, Coordinates are inferred from the ordering and image dimensions; they are not stored separately for every Pixel.
2.3 Pixel Index and Zero-Based Indexing
If we enumerate Pixels from left to right in each row, and from top to bottom between rows, the sequence is:
1 | Pixel: A B C D E F |
The Pixel Index starts at 0. It equals the number of Pixels that precede the selected Pixel in this sequence.
1 | Pixel Index = y * Width + x |
For Pixel A:
1 | x = 0 |
For Pixel E:
1 | x = 1 |
The everyday ordinal number starts at 1:
| Pixel | Coordinates | Pixel Index | Ordinal Number |
|---|---|---|---|
| A | (0, 0) | 0 | 1 |
| B | (1, 0) | 1 | 2 |
| C | (2, 0) | 2 | 3 |
| D | (0, 1) | 3 | 4 |
| E | (1, 1) | 4 | 5 |
| F | (2, 1) | 5 | 6 |
1 | Ordinal Number = Pixel Index + 1 |
3. Raw Image Storage
3.1 Row-Major Layout and Byte Offsets
To store the two-dimensional grid in memory, we need a layout. The left-to-right, top-to-bottom ordering used above is called Row-Major Order.
1 | 2D Image: Pixel Sequence: |
Assume RGB8, with the R, G, and B bytes of each Pixel stored together in that order. Each Pixel occupies 3 bytes. Assume the rows are Tightly Packed, with no extra bytes between rows.
1 | Pixel: A B C |
A Byte Offset measures the distance in bytes from the start of the Pixel Data. It starts at 0, just like Pixel Index, but uses a different unit.
For this tightly packed RGB8 layout:
1 | Bytes per Pixel = 3 |
Pixel E is the fifth Pixel, with Pixel Index 4:
1 | Byte Offset = 4 * 3 = 12 |
3.2 Image Description
Suppose we receive a sequence of 18 bytes:
1 | Byte_0 Byte_1 Byte_2 ... Byte_15 Byte_16 Byte_17 |
The bytes alone do not tell us how to interpret them as an image. Even if we know that they represent six RGB8 Pixels, we still need the dimensions:
1 | Width = 3, Height = 2 |
We also need the Channel Order. The same three bytes have different meanings under RGB and BGR:
1 | Bytes: 255, 0, 0 |
For the simple layout introduced here, an Image Description can specify:
1 | Image Description: |
The Pixel Data provides the values. The Image Description provides the rules for interpreting their layout and Channels. These rules may be provided separately by a program rather than stored inside the Pixel Data itself.
3.3 Stride and Padding
For the tightly packed 3 * 2 RGB8 image:
1 | Row 0: [ A ][ B ][ C ] |
The first row starts at Byte Offset 0, and the second starts at Byte Offset 9. The distance between the starts of consecutive rows is the Stride, measured in bytes.
Some hardware or image-processing interfaces require rows to align to specific byte boundaries.
For example, suppose each row must occupy a multiple of 4 bytes. The Pixel Data in a row occupies 9 bytes, so we can add 3 bytes of Padding to make the Stride 12 bytes:
1 | Pixel Data Padding |
1 | Pixel Data per Row = 9 bytes |
Padding occupies memory but does not represent additional Pixels. Whether Padding is present depends on the layout; it cannot be assumed from Width alone.
For the row-major layout used here, with each Pixel’s Channels stored together:
1 | Byte Offset = y * Stride + x * Bytes per Pixel |
For Pixel E at (1, 1):
1 | Byte Offset = 1 * 12 + 1 * 3 |
For tightly packed rows,
Stride = Width * Bytes per Pixel, which gives the earlier
formula.
3.4 RGBA8 and Alpha
RGB describes a Pixel’s color. When placing an image over a background, we may also need to describe how opaque each Pixel is. RGBA adds an Alpha Channel for that purpose.
For example, a red icon on a blue background may have fully opaque, partially transparent, and fully transparent areas.
In RGBA8, each of the four Channels uses 8 bits:
| Alpha Value | Meaning |
|---|---|
| 255 | Completely Opaque |
| 128 | Approximately 50% Opaque |
| 0 | Completely Transparent |
1 | (255, 0, 0, 255) Fully opaque |
Placed on an opaque blue background:
- Alpha 255: the foreground red completely covers the background.
- Alpha 128: both foreground red and background blue affect the final color.
- Alpha 0: the foreground contributes no color, so the background blue is visible.
These examples describe the RGB color separately from Alpha. Alpha expresses opacity; it is not another color component.
1 | RGB8 Pixel: |
For tightly packed RGBA8:
1 | Bytes per Pixel = 4 |
If the actual Stride is given:
1 | Byte Offset = y * Stride + x * 4 |
3.5 Gray8
Pixel Formats depend on the information we need to represent. RGB describes color, and RGBA adds opacity. If we only need black-to-white levels, a single Channel can describe each Pixel.
In common RGB representations, equal R, G, and B values produce black, white, or a neutral gray:
| RGB Value | Appearance |
|---|---|
| (0, 0, 0) | Black |
| (64, 64, 64) | Darker Gray |
| (128, 128, 128) | Gray |
| (192, 192, 192) | Lighter Gray |
| (255, 255, 255) | White |
For such a black-to-white representation, we can store one Gray Value per Pixel instead of three color components.
In Gray8, the Channel uses 8 bits, so each Pixel occupies 1 byte:
1 | Gray Value: 0 ........ 128 ........ 255 |
Intermediate values indicate different gray levels. These are encoded values; they are not necessarily proportional to physical light output.
3.6 Raw Data Size
For the 8-bit-per-Channel formats discussed above:
| Pixel Format | Channels | Bytes per Pixel |
|---|---|---|
| Gray8 | Gray | 1 |
| RGB8 | R, G, B | 3 |
| RGBA8 | R, G, B, A | 4 |
For tightly packed rows:
1 | Pixel Count = Width * Height |
For example:
| Pixel Format | Pixel Dimensions | Raw Data Size |
|---|---|---|
| Gray8 | 1920 * 1080 | 2,073,600 bytes |
| RGB8 | 1920 * 1080 | 6,220,800 bytes |
| RGBA8 | 1920 * 1080 | 8,294,400 bytes |
If every row, including the last, is allocated a complete Stride of storage:
1 | Allocated Size = Stride * Height |
For the padded 3 * 2 RGB8 example:
1 | Pixel Data = 3 * 2 * 3 = 18 bytes |
Some interfaces do not require complete trailing Padding after the last row, so the allocation must follow the actual interface’s requirements.
These calculations describe Raw Pixel Data or its memory allocation. An encoded image file can have a different size.
4. Image Compression
4.1 Redundancy
Suppose an image has Pixel Dimensions 8 * 1, and every
Pixel is red:
1 | Width = 8 |
Each Pixel has RGB value (255, 0, 0). Tightly packed Raw
Pixel Data occupies 8 * 3 = 24 bytes.
We can describe the same sequence as Repeat Red 8 times. This uses two pieces of information:
1 | Color: (255, 0, 0) |
This repeated content is a simple example of exploitable Redundancy.
To store the rule, define a simple teaching format:
1 | Record Format: |
The Record occupies 4 bytes, compared with the original 24 bytes. This comparison counts only the Record and Raw Pixel Data, excluding other description information; the reader must already know the Encoding rule.
During Decoding, the reader interprets
(8, 255, 0, 0) as eight consecutive Pixels with RGB value
(255, 0, 0).
1 | Raw Pixel Data |
The Pixel Count and RGB values are preserved. The representation uses fewer bytes.
4.2 Lossless Compression: RLE
The previous example introduces the idea behind Run-Length Encoding (RLE). Now consider multiple colors:
1 | [Red][Red][Red][Blue][Blue][Green][Green][Green] |
A Run is a consecutive sequence of Pixels with the same color:
1 | Run 1: Red * 3 |
Each Run has a corresponding Record. Using the same 4-byte Record Format:
| Run | Count | R | G | B |
|---|---|---|---|---|
| 1 | 3 | 255 | 0 | 0 |
| 2 | 2 | 0 | 0 | 255 |
| 3 | 3 | 0 | 255 | 0 |
In this teaching format, Count uses values from 1 to 255. A longer Run can be split across multiple Records.
1 | Raw Data Size = 8 * 3 = 24 bytes |
To Decode the Records:
1 | Record 1 -> Red Red Red |
Every original Pixel Value is recovered exactly. This is Lossless Compression.
However, RLE can increase the size when Runs are short. For example:
1 | Red Green Red Green Red Green Red Green |
Although each color appears repeatedly, no adjacent Pixels have the same color. Each Run has Count 1, so each Pixel gains an extra Count byte.
RLE’s effectiveness depends on the length of consecutive repetition, rather than just the total number of appearances of a color. This is a limitation of this RLE scheme, not of every Lossless Compression method.
4.3 Lossy Compression: Quantization
The RLE scheme above benefits from identical neighboring Pixel Values. If neighboring colors are close but not identical, we can ask what happens if we allow some differences to be discarded.
Consider eight RGB8 Pixels:
1 | Pixel: A B C D E F G H |
These Pixels form eight Runs in the RLE scheme. Their colors are close, although whether differences are noticeable depends on the content and viewing conditions.
If we represent these similar colors using a common value, can we obtain a longer Run?
For each Channel independently, group the original values into groups of eight adjacent values and choose the first value as the representative. This is a simple example of Quantization:
1 | Original Channel Values Group Index Quantized Channel Value |
The rule maps multiple Channel Values to one representative value. Applied to the example:
1 | Pixel: A B C D E F G H |
If we still store every Pixel as RGB8, Quantization alone does not reduce the byte count. But the resulting single Run can now be represented by one RLE Record:
1 | Count R G B |
| Representation | Data Size |
|---|---|
| Original RGB8 | 24 bytes |
| Original RGB8 + RLE | 32 bytes |
| Quantized RGB8 | 24 bytes |
| Quantized RGB8 + RLE | 4 bytes |
The smaller result comes from Quantization + RLE. Quantization changes the values; RLE then represents the repeated result compactly.
After Decoding, all eight R values are 248:
1 | Original R: |
A decoded value of 248 no longer identifies which original value produced it. Information has been lost, making this a teaching example of Lossy Compression.
Lossy Compression does not guarantee that the changes are invisible. Whether the loss is acceptable depends on the image and its intended use.
4.4 Encoded Size and Quality
The previous example reduces Encoded Size while keeping the same number of Pixels. Several different properties must therefore be distinguished:
| Property | Meaning |
|---|---|
| Pixel Dimensions | Width and Height in pixels |
| Raw Data Size | Bytes used by the chosen raw Pixel Format and layout |
| Encoded Data Size | Bytes used by the encoded representation |
| Quality | Fidelity to the original content and the resulting visual appearance |
For the eight-Pixel example from Section 4.3:
| Representation | Pixel Dimensions | Data Size | Exact Recovery |
|---|---|---|---|
| Raw RGB8 | 8 * 1 | 24 bytes | Yes |
| RLE | 8 * 1 | 32 bytes | Yes |
| Quantization + RLE | 8 * 1 | 4 bytes | No |
The RLE representation is larger than Raw RGB8 but recovers exactly the same content. More bytes do not necessarily mean higher Quality.
The Quantization + RLE representation keeps the same Width and Height but changes Pixel Values. Equal Pixel Dimensions do not guarantee equal Quality.
When the 4-byte representation is decoded into tightly packed RGB8,
it again requires 8 * 3 = 24 bytes to hold the eight
decoded Pixels. Encoded Data Size and decoded Raw Data Size describe
different representations.
The comparisons above count the data representation itself. Image files also need a defined way to organize their information, which is the next topic.
5. Image File Formats
5.1 Raw Pixel Data vs Image Files
A 8 consecutive red pixel in RLE record:
1 | Count R G B |
If a image process application get the data, it doesn’t know what it is.
- What are the four numbers?
- The format is
8 * 1or4 * 2?