A deep dive into images
on the web

and then some…

Chen Hui Jing / @hj_chen

Chen Hui Jing
Jing
@hj_chen
🇲🇾 🏀 👩‍💻 📝 🦊
Nexmo Developer Relations

🥑 Developer Advocate 🥑

Nexmo

Electronic signals

Pixel
Etymology of Pixel
CRT
CRTs sold by Müller-Uri, Source: The Cathode Ray Tube site

Raster scanning

Raster scan
Source: M-SYS MV
Raster scan Random scan
Electron beam Swept across entire screen, one row at a time, from top-to-bottom Only directed to parts of the screen where image is drawn
Resolution Poor, due to plotting as discrete point sets Good, as CRT beam directly follows line path
Picture definition Stored as set of intensity values (pixels) in refresh buffer area Stored as set of line drawing instructions in display file
Realism Variable intensity values allow for realistic shadow and colour patterns Most suited for line drawing
Drawing method Screen points (pixels) Mathematical functions

Source: Prof. Vijay M. Shekhat, CE Department, Computer Graphics

File formats

. . . File extension External metadata Mac OS type-codes Mac OS X UTIs OS/2 extended attr. POSIX extended attr. PUIDs MIME types FFIDs File content-based identification Internal metadata File header / Magic number

Colour depth

1-bit image
1-bit PNG
(2 colours)
2-bit image
2-bit PNG
(4 colours)
4-bit image
4-bit PNG
(16 colours)
8-bit image
8-bit PNG
(256 colours)
24-bit image
24-bit PNG
(16,777,216 colours)

Source: Wikipedia, Color depth

MacPaint

Screenshot I of MacPaint Screenshot II of MacPaint

MS Paintbrush

MS Paintbrush on Windows 3.0
Paintbrush for Windows 3.0
MS Paintbrush on Windows 3.1
Paintbrush for Windows 3.1

Bitmap (BMP)

Palette-indexed bitmap
Bitmap and its corresponding colour table
Direct colour storage
Colours directly in bitmap itself

Source: Microsoft, Types of Bitmaps

Graphics Interchange Format (GIF)

GIF specification

Source: Graphics Interchange Format (tm)

Compression algorithms

Run length compression used by MacPaint
Lempel–Ziv–Welch (LZW) compression

Motion graphics by Crystal Law

Les Horribles Cernettes
First band photo on the web

Source: The Cernettes

Joint Photographic Experts Group (JPEG)

JPEG

JPG compression

JPG compression

Reference: How JPG works

JPG compression

Reference: How JPG works

JPG compression

Reference: How JPG works

JPG compression

Reference: How JPG works

JPG compression

Reference: How JPG works

JPG compression

Reference: How JPG works

Progressive JPGs

Timeline of loading progressive jpg
Image source: What is a progressive JPEG?

JPG optimisation tips

  1. Use high quality source material
  2. Alignment on the 8x8 pixel grid
  3. Reduce contrast and saturation
  4. Sepia images
  5. Slight blurring

Reference: Finally understanding JPG

2. Alignment on the 8x8 pixel grid

Alignment to 8x8 grid Misaligned from grid altogether

Reference: Finally understanding JPG

3. Reduce contrast and saturation

LAB colour mode in Affinity Photo

Reference: Finally understanding JPG

4. Sepia images

Una Kravets at JSConf.Asia 2018
Una Kravets: CSS Blend Modes, Because…

Reference: Finally understanding JPG

5. Slight blurring

Non-key areas of image blurred out No blur applied

Reference: Finally understanding JPG

Speed, Quality, Size

Can't have 'em all

JPG encoders, there are many

libjpeg
libjpeg
mozJPEG
mozJPEG
Guetzli
Guetzli

libjpeg-turbo

libjpeg-turbo home page
https://libjpeg-turbo.org/

GIF89a

Burn All GIFs day

November 5, 1999

https://burnallgifs.org/archives/

Portable Network Graphics (PNG)

PNG header
8-byte signature of a PNG file

PNG filter algorithms

None Zero None
Sub Sub(x) = Raw(x) - Raw(x-bpp) Byte A (to the left) Sub
Up Up(x) = Raw(x) - Prior(x) Byte B (above) Up
Average Average(x) = Raw(x) - floor((Raw(x-bpp)+Prior(x))/2) Mean of bytes A and B, rounded down Average
Paeth Paeth(x) = Raw(x) - PaethPredictor(Raw(x-bpp), Prior(x), Prior(x-bpp)) A, B, or C, whichever is closest to
p = A + B − C
Paeth

DEFLATE compression algorithm

DEFLATE compression algorithm specification
DEFLATE Compressed Data Format Specification version 1.3

What a difference 2 pixels can make

Difference in compression rates of 2 sets of images of kiwis

Reference: pngthermal

Optimising PNG files

  1. Reduce number of colours
  2. Choose the right pixel format
  3. Use indexed images, if possible
  4. Optimise fully transparent pixels

Reference: Reducing PNG file Size

3. Use indexed images, if possible

Full colour PNG
Full colour PNG
Indexed palette PNG
Indexed colour PNG

Reference: Reducing PNG file Size

4. Optimise fully transparent pixels

Masked portions of image in single colour
Masked portion of image filled with single colour
Masked portions of image unoptimised
Masked portion of image untouched

Reference: Reducing PNG file Size

libpng

libpng home page
http://www.libpng.org/pub/png/libpng.html

What is a browser engine? 🤔

The blackbox of browser engines

Source: Quantum Up Close: What is a browser engine?

What is a browser engine? 🤓

Browser engine diagram

Source: Quantum Up Close: What is a browser engine?

Image encoders in browsers

Chromium GitHub repository
Chromium
Geckto GitHub repository
Gecko

Browser rendering pipeline

Layout computation to frame tree

Reference: Introduction to WebRender – Part 1 – Browsers today

Browser rendering pipeline

Generation of display list

Reference: Introduction to WebRender – Part 1 – Browsers today

Browser rendering pipeline

Painting display list to layers

Reference: Introduction to WebRender – Part 1 – Browsers today

Browser rendering pipeline

Layers composited into final image

Reference: Introduction to WebRender – Part 1 – Browsers today

How to build a browser engine series

Reference: Let's build a browser engine!

Rasterisation (1/3)

Simple rasteriser for painting rectangles

pub struct Canvas {
    pub pixels: Vec<Color>,
    pub width: usize,
    pub height: usize,
}

impl Canvas {
    /// Create a blank canvas
    fn new(width: usize, height: usize) -> Canvas {
        let white = Color { r: 255, g: 255, b: 255, a: 255 };
        Canvas {
            pixels: vec![white; width * height],
            width: width,
            height: height,
        }
    }
  // …
}

Source: Let's build a browser engine!

Rasterisation (2/3)

Simple rasteriser for painting rectangles

fn paint_item(&mut self, item: &DisplayCommand) {
    match *item {
        DisplayCommand::SolidColor(color, rect) => {
            // Clip the rectangle to the canvas boundaries.
            let x0 = rect.x.clamp(0.0, self.width as f32) as usize;
            let y0 = rect.y.clamp(0.0, self.height as f32) as usize;
            let x1 = (rect.x + rect.width).clamp(0.0, self.width as f32) as usize;
            let y1 = (rect.y + rect.height).clamp(0.0, self.height as f32) as usize;

            for y in y0 .. y1 {
                for x in x0 .. x1 {
                    self.pixels[y * self.width + x] = color;
                }
            }
        }
    }
}

Source: Let's build a browser engine!

Rasterisation (3/3)

Simple rasteriser for painting rectangles

/// Paint a tree of LayoutBoxes to an array of pixels.
pub fn paint(layout_root: &LayoutBox, bounds: Rect) -> Canvas {
    let display_list = build_display_list(layout_root);
    let mut canvas = Canvas::new(bounds.width as usize, bounds.height as usize);
    for item in display_list {
        canvas.paint_item(&item);
    }
    canvas
}

Source: Let's build a browser engine!

The graphics backend is an array of bytes and a for loop that goes and says: go from the left edge of the rectangle to the right edge and write the same colour over and over again into this array.


—Matt Brubeck, Bay Area Rust Meetup (Nov 2014)
¯\_(ツ)_/¯

Graphics libraries

skia logo
Skia
Mozilla GFX team logo
WebRender
Mozilla GFX team logo
Core Graphics
cairo logo
Cairo
pango logo
Pango

Painting

Mapping of colour information from frame buffer to bitmap

Reference: The whole web at maximum FPS: How WebRender gets rid of jank

Compositing (1/2)

Compositing process for scrolling a web page

Reference: The whole web at maximum FPS: How WebRender gets rid of jank

Compositing (2/2)

Figure showing only layers with changes get invalidated

Reference: GPU Accelerated Compositing in Chrome

Making use of the GPU (1/2)

CPU cores versus GPU cores

Making use of the GPU (2/2)

Reference: Hardware acceleration and compositing

GPU rasterisation in Chromium

Reference: Software vs. GPU rasterization in Chromium

If you could have a do-over…

Seedling growing into a flower
What if we actually need a butterfly instead?

WebRender (1/2)

Drawing of a jet engine labeled with the different Project Quantum projects
The whole web at maximum FPS: How WebRender gets rid of jank by Lin Clark

WebRender (2/2)

Acknowledgements

🙏 Big thank you to these beautiful human beings who answered my noob questions 🙏

References

This talk is dedicated to browser engineers everywhere.

I owe my career to you.

Thank you!

Websitehttps://www.chenhuijing.com

Twitter@hj_chen

Medium@hj_chen

Codepen@huijing

Font used is Signika by Anna Giedryś