1. Introduction

In computer science area, we can deem that the basic information component is the bit streams representing by 0, 1 string. Based on 0 and 1 bit streams, we have developed information patterns such as text, images, videos, and audio that can be intuitively understood by humans.

Text

Image

1. RGB Principal

1. The visual mechanism of the human eye provides a basis for describing color using three quantities.

Light possesses various spectral compositions. Upon entering the eye, light stimulates photoreceptor cells, and the visual system subsequently generates the perception of color based on these stimuli.

In humans with normal vision, photopic (daylight) vision relies primarily on three types of cone cells, each exhibiting a distinct response to different ranges of light wavelengths. Consequently, under specific viewing conditions, the stimulation of these three types of cells by a given light source can be characterized by three distinct quantities.

This implies that even if two light sources have different spectral compositions, they can still be perceived as the same color, provided they elicit identical levels of stimulation across these three types of cone cells.

This principle serves as the foundation for display devices to reproduce a wide range of colors using only a limited number of light sources.

2. A specific range of colors can be reproduced by adjusting red, green, and blue light.

By selecting red, green, and blue as primary colors and independently controlling the intensity of each, display devices can generate a diverse array of color perceptions.

For instance, an appropriate mixture of red and green light is perceived by the human eye as yellow.

Thus, any color that can be reproduced using this set of primary colors can be described by three individual components:

  • Red Component
  • Green Component
  • Blue Component

3. Color Representation in Space Distribution

A picture in computer can be divided into the minimized unit: Pixel. We can simply deem a pixel is a point.

A picture is composed of many pixels.

We need to describe two things:

  • Pixel Location
  • Pixel Color

So a picture can be described as Pixel Location and Color information.

4. Color Component Precision

We have discussed above that three basic color components R, G and B can be used to compose different other colors. For each color component, there is a difference in color degree. For example, there is a difference between light red and dark red. In the computer, we divide the degree of distinction of each color component into an interval from 0 to 255.

1
2
3
R: 0 ~ 255
G: 0 ~ 255
B: 0 ~ 255

Now a pixel’s color can represent as:

1
(255, 0, 0)

In a given RGB color space, 255 represents the maximum encoded value for a channel, while 0 represents a value of zero for that channel.

5. Color Precision Represented by Bit

Obviously, we can use 8 bits to represent 0 ~ 255.

e.g.

1
2
3
R          G          B
255 0 0 (Decimal)
11111111 00000000 00000000 (Binary)

2. From Pixel to Image (Raster Image)

We have discussed the RGB principle and the image’s basic unit: pixel.

An image is in fact a 2-dimension pixel grid.

e.g. There are 6 pixels from A to F with corresponding RGB value arranging in 3 columns and 2 rows.

1
2
3
4
5
6
7
8
9
10
            x →
0 1 2
+-----+-----+-----+
y 0 | A | B | C |
↓ +-----+-----+-----+
1 | D | E | F |
+-----+-----+-----+

Width = 3 pixels
Height = 2 pixels

Common Conventions:

  • The upper left corner is the origin
  • x increases to the right
  • y increases downward

So the pixel E with coordinates (1, 1).

This image described by 2-dimensions pixel grid is called Raster Image.

So each pixel contains two information entry, the position and color:

1
2
Position: (x, y)
Color: (R, G, B)

3. Image Storage

3.1 From Image to Bits

How to store an image in the computer?

1
2
3
4
5
6
7
          x →
0 1 2
+-----+-----+-----+
0 | A | B | C |
y +-----+-----+-----+
↓ 1 | D | E | F |
+-----+-----+-----+

A common way is: from left to right in each row, and from top to bottom between rows. This is called row-major order.

1
2
3
4
2D Image:          Pixel Sequence:

A B C A → B → C → D → E → F
D E F

Assume that each pixel has three color channels (RGB) and each channel takes 8 bits. So a single pixel needs 3B space. 6 pixels take 3 * 6 = 18B in total.

1
2
3
4
5
6
7
Pixel:    A           B           C
Bytes: R_A G_A B_A R_B G_B B_B R_C G_C B_C
Offset: 0 1 2 3 4 5 6 7 8

Pixel: D E F
Bytes: R_D G_D B_D R_E G_E B_E R_F G_F B_F
Offset: 9 10 11 12 13 14 15 16 17

Offset refers to the byte offset relative to the start of this pixel data, starting from 0.

So if we have the image’s binary information . If we want to know the 5th pixel’s color, we can infer that the 5th pixel’s offset number is 3 * (5 - 1) = 12.

1
2
3
4
5th Pixel(E):

Offset: 12 13 14
Channel: R G B

General Formula for Row-Majored Order:

1
2
3
Pixel index = y × Width + x

Byte offset = Pixel index × 3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
                  x
0 1 2
+-----+-----+-----+
0 | A | B | C |
y +-----+-----+-----+
1 | D | E | F |
+-----+-----+-----+

Width = 3
Height = 2

Pixel: A B C D E F
Index: 0 1 2 3 4 5

  • X-Coordinate starts from zero (0, 1, 2 … width - 1)
  • Y-Coordinate starts from zero (0, 1, 2 … height - 1)

Pixel A:

1
2
3
4
5
6
7
x = 0
y = 0
width = 3

Pixel index = y × Width + x = 0 × 3 + 0 = 0

Byte offset = Pixel index × 3 = 0 × 3 = 0

Pixel E:

1
2
3
4
5
6
7
x = 1
y = 1
width = 3

Pixel index = y × Width + x = 1 × 3 + 1 = 4

Byte offset = Pixel index × 3 = 4 × 3 = 12

3.2 From Bits to Image

Suppose that we have a string of RGB bitstreams:

1
B_0 B_1 B_2 ... B_12 B_13 B_14 B_15

Can we restore the image ? Of course not, because

  1. We do not know the space information of the image. e.g. 6 pixels can be arranged as 3 * 2 or 2 * 3.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    Width = 3, Height = 2

    A B C
    D E F

    Width = 2, Height = 3

    A B
    C D
    E F
  2. We don’t know the RGB rule. The three color bytes can be described into different rules

1
255, 0, 0

RGB rule:

1
2
3
R = 255
G = 0
B = 0

BGR rule:

1
2
3
B = 255
G = 0
R = 0

So we need extra information to record full information:

1
2
3
4
5
6
7
8
9
10
Image description:
Width: 3
Height: 2
Channel order: RGB
Bits per channel: 8
Pixel order: Row-major
Packing: Tightly packed

Pixel data:
R_A G_A B_A R_B G_B B_B ... R_F G_F B_F

3.3 Stride

For a 3 * 3 image:

1
2
3
4
5
6
7
8
Row 0:  [ A ][ B ][ C ]
Row 1: [ D ][ E ][ F ]

Pixel: A B C
Offset: 0 3 6

Pixel: D E F
Offset: 9 12 15

The gap of the offset of each line is 9 which is called Stride.

However some hardware or image processing interfaces require row data to be aligned to specific byte boundaries.

Suppose each row is required to take up space in multiples of 4 bytes. A row of color data has 9 bytes, so you can add 3 bytes at the end to make it 12 bytes:

1
2
3
4
5
6
7
8
9
10
11
12
         Pixel data       Padding
+---+---+---+----------------+
Row 0: | A | B | C | 3 bytes |
+---+---+---+----------------+
Row 1: | D | E | F | 3 bytes |
+---+---+---+----------------+

Pixel: A B C
Offset: 0 3 6

Pixel: D E F
Offset: 12 15 18
1
2
3
Pixel data per row = 9 bytes
Padding per row = 3 bytes
Stride = 12 bytes

Formula for Row-Majored Order with Stride:

1
Byte offset = y × Stride + x × Bytes per pixel

e.g. the pixel E (1,1)’s offset:

1
offset = 1 * 12 + 1 * 3 = 15

3.4 RGBA (Transparency)

RGBA is an extension format of RGB with an extra channer Alpha to describe the transparency.

Why we need Alpha to describe the transparency?

Suppose we want to place a red icon on a blue background. The icon might have areas that completely cover the background, or it might have transparent or semi-transparent areas.

RGB can only describe the color of a pixel; it can’t tell us how it should blend with the background on its own.

Alpha Value Meaning
255 Completely Opaque
128 Half Opaque
0 Completely Transparent

For instance, a red pixel can have different alpha value:

1
2
3
(255, 0, 0, 255)   Fully opaque
(255, 0, 0, 128) Partially transparent
(255, 0, 0, 0) Fully transparent

Place the red pixels on a blue background: - Alpha 255: The spot shows the foreground red, completely covering the background. - Alpha 128: The foreground red and background blue both affect the final color. - Alpha 0: The foreground doesn’t contribute color, so the spot shows the background blue.

With an extra channel, the pixel’s color information will increase from 3 to 4 bytes.

1
2
3
4
5
6
7
8
9
RGB pixel:
+---+---+---+
| R | G | B | 3 bytes
+---+---+---+

RGBA pixel:
+---+---+---+---+
| R | G | B | A | 4 bytes
+---+---+---+---+

So the formula for RGBA8 (8 for 8 bits):

1
2
Bytes per pixel = 4
Byte offset = (y × Width + x) × 4

If the stride is known:

1
Byte offset = y × Stride + x × 4

3.5 Grayscale Image

As we said above, we can use RGB combination to illustrate a specific range of colors. There’s a phenomenon: when R = G = B, we will get black, white and gray.

RGB value Appearance
(0, 0, 0) Black
(64, 64, 64) Darker Gray
(128, 128, 128) Gray
(192, 192, 192) Lighter Gray
(255, 255, 255) white

For this kind image, we can use single-channel to represent the color instead of using three kinds.

Grayscale pixels usually only keep one gray value.

Using the common 8-bit grayscale format, usually called Gray8:

1
2
Gray value:  0 ........ 128 ........ 255
Black Gray White

Each pixel only includes a gray value:

  • 0 for black
  • 255 for white
  • middle values indicate different shades of gray

4. Image Compression

4.1 Redundancy

Suppose an image’s specification is 1 * 8, and each pixel is red.

1
2
3
4
5
Width = 8
Height = 1
Pixel Format = RGB8

[Red][Red][Red][Red][Red][Red][Red][Red]

So each pixel’s RGB value is: (255, 0, 0). Totally cost 3 * 8 = 24 Bytes

We can use one sentence to describe these pixels: Reapt Red 8 times. This rule includes following information:

1
2
Color: (255, 0, 0)
Count: 8

Reapt Red 8 times is said for human. In computer, we can use the following format data to record consecutive identical value:

  • Format:

    1
    2
    3
    +---------------+--------------+--------------+-----------+
    | Count: 1 byte | R Value: 1B | G Value: 1B | B Value: 1B |
    +---------------+--------------+--------------+-----------+
  • Example:

    1
    2
    Count    R     G     B
    8 255 0 0

    This record only takes 4B space comparing with the original 24B.

Decompress:

When the image dealing process read: (8, 255, 0, 0), it knows that there are 8 consecutive (255, 0, 0)

5. Image File Formats

6. Image Processing

7. Image Display

Audio