A deep dive into images
on the web

and then some…

Hello! This is my very first Fronteers ever and I'm really happy to be here. A big thank you to the organisers for having me this year. Today, I'd like to share with everyone my journey of trying to figure out images on the web. You see, I have this tendency to go down rabbit holes with things that seem rather straight-forward at first, but then, I start to wonder if there's something more beneath the surface?

So take this as a bit of a content warning, that this talk will go places.

But anyway, that's what happened to me with images. I'm also very annoying and have an incessant need to ask why. Constantly. Which is why I love and appreciate everyone who is still willing to be friends with me. I've been working on the web for a number of years now, and I learned that image optimisation is one of the low-hanging fruits when it comes to web performance.

🇲🇾 👾 🏀 🚲 🖌️ 👟 💻 🖊️ 🎙 🦊 🥑 🧗 🏳️‍🌈

@hj_chen

Electronic signals

CRTs sold by Müller-Uri, Source: The Cathode Ray Tube site

Raster scanning

	Raster scan	Random scan
Electron beam	Swept across entire screen, one row at a time, from top-to-bottom	Only directed to parts of the screen where image is drawn
Resolution	Poor, due to plotting as discrete point sets	Good, as CRT beam directly follows line path
Picture definition	Stored as set of intensity values (pixels) in refresh buffer area	Stored as set of line drawing instructions in display file
Realism	Variable intensity values allow for realistic shadow and colour patterns	Most suited for line drawing
Drawing method	Screen points (pixels)	Mathematical functions

Source: Prof. Vijay M. Shekhat, CE Department, Computer Graphics

File formats

Colour depth

24-bit image — 24-bit PNG
(16,777,216 colours)

Source: Wikipedia, Color depth

MacPaint

MS Paintbrush

Bitmap (BMP)

Palette-indexed bitmap — Bitmap and its corresponding colour table

Direct colour storage — Colours directly in bitmap itself

Source: Microsoft, Types of Bitmaps

Graphics Interchange Format (GIF)

Source: Graphics Interchange Format (tm)

Compression algorithms

Run length compression used by MacPaint

Lempel–Ziv–Welch (LZW) compression

Motion graphics by Crystal Law

Les Horribles Cernettes — First band photo on the web

Source: The Cernettes

Joint Photographic Experts Group (JPEG)

JPG compression

Reference: How JPG works

Progressive JPGs

Timeline of loading progressive jpg — Image source: What is a progressive JPEG?

JPG optimisation tips

Use high quality source material
Alignment on the 8x8 pixel grid
Reduce contrast and saturation
Sepia images
Slight blurring

Reference: Finally understanding JPG

2. Alignment on the 8x8 pixel grid

Reference: Finally understanding JPG

3. Reduce contrast and saturation

Reference: Finally understanding JPG

4. Sepia images

Una Kravets at JSConf.Asia 2018 — Una Kravets: CSS Blend Modes, Because…

Reference: Finally understanding JPG

Another pretty smart trick is to get rid of the colour information altogether, then colourising the image via CSS, with properties like filter or even in combination with blend modes. If blend modes are your thing, you must check out Una Kravet's amazing talk on blend modes, link in the slide.

5. Slight blurring

Reference: Finally understanding JPG

Speed, Quality, Size

Can't have 'em all

JPG encoders, there are many

libjpeg-turbo

GIF89a

Burn All GIFs day

November 5, 1999

https://burnallgifs.org/archives/

Portable Network Graphics (PNG)

PNG header — 8-byte signature of a PNG file

They came up with a little something called Portable Network Graphics, shortened from the proposed PING, for “Ping is not GIF”. Some other suggestions were quite fun, like TNT, for “The New Thing”.

PNGs outperformed GIFs in a multitude of ways. It was 100% open-source and patent-free. It could support thousands of colours versus GIF's 256. It handled transparency better.

But the designers of PNG made a conscious decision not to provide animation capabilities. And that was pretty much what kept the GIF alive all these years, because there was no viable alternative to animated GIFs.

In contrast to JPG, PNG compression is lossless. And it is a 2-part process. Delta encoding is a way of storing or transferring data via differences between sequential data instead of complete files. Diff-ing, essentially.

PNG filter algorithms

None	—	Zero
Sub	`Sub(x) = Raw(x) - Raw(x-bpp)`	Byte A (to the left)
Up	`Up(x) = Raw(x) - Prior(x)`	Byte B (above)
Average	`Average(x) = Raw(x) - floor((Raw(x-bpp)+Prior(x))/2)`	Mean of bytes A and B, rounded down
Paeth	`Paeth(x) = Raw(x) - PaethPredictor(Raw(x-bpp), Prior(x), Prior(x-bpp))`	A, B, or C, whichever is closest to `p = A + B − C`

DEFLATE compression algorithm

What a difference 2 pixels can make

Difference in compression rates of 2 sets of images of kiwis

Reference: pngthermal

Specifically, how the exact dimensions of your image could have an impact on how well image compression works. If you look at the illustration, you'll see two nearly identical images of kiwi fruits. Only that the image on the right is 2 pixels wider.

For the smaller image, the bytes representing the second and third kiwis fell between the LZ match range and are encoded as highly efficient LZ matches, hence the dark blue thermal pattern.

But somehow, the extra 2 columns of pixels in the right image resulted in the bytes for the second and third kiwis to be out of match range, and they are encoded as non-matching. Thus causing the file size to bloat dramatically, doubling in size.

PNGs are a regularly-used format on the web, so let's go through some tips to keep their file sizes down as well.

Optimising PNG files

Reduce number of colours
Choose the right pixel format
Use indexed images, if possible
Optimise fully transparent pixels

Reference: Reducing PNG file Size

Again, if you can, use a tool. Automate all the things. But if you had to do some manual tweaking, one way to reduce the file size of your PNGs is by reducing the number of unique colours within the image.

Because this impacts the filtering stage, where a lower difference between adjacent pixels will decrease the range of values output from the filtering process. With more duplicate values, DEFLATE can compress better.

HOWEVER, this makes it a lossy process, which is why your human eye is absolutely required if you want to take this approach. Our current tools are still unable to discern visual quality at a level acceptable to human perception. For now.

Also, if you're not using transparency in your image, then don't bother with RGBA 32 bits per pixel. You could use the 24 bits per pixel truecolour format instead, or even just use a JPG, why not? For greyscale images, 8 bits per pixel will suffice.

3. Use indexed images, if possible

Indexed palette PNG — Indexed colour PNG

Reference: Reducing PNG file Size

4. Optimise fully transparent pixels

Masked portions of image in single colour — Masked portion of image filled with single colour

Masked portions of image unoptimised — Masked portion of image untouched

Reference: Reducing PNG file Size

libpng

What is a browser engine? 🤔

Source: Quantum Up Close: What is a browser engine?

What is a browser engine? 🤓

Source: Quantum Up Close: What is a browser engine?

Image encoders in browsers

Browser rendering pipeline

Reference: Introduction to WebRender – Part 1 – Browsers today

Nicolas Silva, who leads the Firefox GFX team, was a great help as he explained a lot of this stuff to me. Although each browser engine does things slightly differently, the general idea behind a modern rendering pipeline involves:

layout computation into a frame tree
generation of drawing commands called a display list
the painting of portions of that display list into layers
and finally, combining those layers into one final image through compositing

Because the displays that we use now are all raster displays, it is necessary for a rasterisation process to occur before graphics can be displayed on the screen. Decoded images are generally already in a raster format, but vector graphics or fonts need to be expressed as pixels as well.

Browser rendering pipeline

Reference: Introduction to WebRender – Part 1 – Browsers today

layout computation into a frame tree
generation of drawing commands called a display list
the painting of portions of that display list into layers
and finally, combining those layers into one final image through compositing

Browser rendering pipeline

Reference: Introduction to WebRender – Part 1 – Browsers today

layout computation into a frame tree
generation of drawing commands called a display list
the painting of portions of that display list into layers
and finally, combining those layers into one final image through compositing

Browser rendering pipeline

Reference: Introduction to WebRender – Part 1 – Browsers today

layout computation into a frame tree
generation of drawing commands called a display list
the painting of portions of that display list into layers
and finally, combining those layers into one final image through compositing

Reference: Let's build a browser engine!

Rasterisation (1/3)

Simple rasteriser for painting rectangles

pub struct Canvas {
    pub pixels: Vec<Color>,
    pub width: usize,
    pub height: usize,
}

impl Canvas {
    /// Create a blank canvas
    fn new(width: usize, height: usize) -> Canvas {
        let white = Color { r: 255, g: 255, b: 255, a: 255 };
        Canvas {
            pixels: vec![white; width * height],
            width: width,
            height: height,
        }
    }
  // …
}

Source: Let's build a browser engine!

Rasterisation (2/3)

Simple rasteriser for painting rectangles

fn paint_item(&mut self, item: &DisplayCommand) {
    match *item {
        DisplayCommand::SolidColor(color, rect) => {
            // Clip the rectangle to the canvas boundaries.
            let x0 = rect.x.clamp(0.0, self.width as f32) as usize;
            let y0 = rect.y.clamp(0.0, self.height as f32) as usize;
            let x1 = (rect.x + rect.width).clamp(0.0, self.width as f32) as usize;
            let y1 = (rect.y + rect.height).clamp(0.0, self.height as f32) as usize;

            for y in y0 .. y1 {
                for x in x0 .. x1 {
                    self.pixels[y * self.width + x] = color;
                }
            }
        }
    }
}

Source: Let's build a browser engine!

Rasterisation (3/3)

Simple rasteriser for painting rectangles

/// Paint a tree of LayoutBoxes to an array of pixels.
pub fn paint(layout_root: &LayoutBox, bounds: Rect) -> Canvas {
    let display_list = build_display_list(layout_root);
    let mut canvas = Canvas::new(bounds.width as usize, bounds.height as usize);
    for item in display_list {
        canvas.paint_item(&item);
    }
    canvas
}

Source: Let's build a browser engine!

The graphics backend is an array of bytes and a for loop that goes and says: go from the left edge of the rectangle to the right edge and write the same colour over and over again into this array.

—Matt Brubeck, Bay Area Rust Meetup (Nov 2014)

¯\_(ツ)_/¯

Graphics libraries

Skia

WebRender

Core Graphics

Cairo

Pango

Of course, commercial browsers make use of graphics APIs and libraries to implement rasterisation, because clearly what we expect from the web is much more than just solid coloured rectangles. Browser makers need functions for rendering text, polygons, lines, gradients, curves…you know, all of the things.

This is a list of common graphics libraries currently being used to power the popular browser engines. Chrome uses the Skia graphics library almost exclusively for all graphics operations, even text rendering.

Firefox used to use CoreGraphics and Cairo but eventually simplified the number of backends they ran on. And after the smoke clears on their big engine overhaul, Firefox will eventually use Skia for canvas, and WebRender for everything else.

Safari, which is based on WebKit, apparently used Apple's Core Image libraries, but I'm not sure what they use right this minute.

Painting

Mapping of colour information from frame buffer to bitmap

Reference: The whole web at maximum FPS: How WebRender gets rid of jank

The concept of painting still remains the same though, with displays accessing the frame buffer for information on every pixel that needs to be displayed onto the screen in RGBA format. A frame is considered rendered when all the pixels have been filled in by the renderer.

This process is constantly being repeated every time something on the page changes. But most of the time, only a part of the screen is changing. Browsers will figure out what changed and only update those relevant pixels. This is called invalidation.

Invalidation as an optimisation technique has been around since the early browsers, but they could only get us so far. Though invalidation techniques work well for small changes, like a blinking caret on an input field, sweeping changes affecting most of the screen required something more.

Compositing (1/2)

Compositing process for scrolling a web page

Reference: The whole web at maximum FPS: How WebRender gets rid of jank

Compositing (2/2)

Figure showing only layers with changes get invalidated

Reference: GPU Accelerated Compositing in Chrome

Making use of the GPU (1/2)

Making use of the GPU (2/2)

Reference: Hardware acceleration and compositing

GPU rasterisation in Chromium

Reference: Software vs. GPU rasterization in Chromium

The Chromium Project has very detailed design documentation which outline exactly what goes on under the hood with Blink and cc, the Chrome compositor.

In Chromium, the page is divided up into tiles of 256 by 256 pixels for more efficient rasterisation. Paint commands which don't impact certain tiles get ignored and only the tiles which need to be updated get rasterised again.

The old way of rasterising a tile makes use of the Skia library, which uses a scanline algorithm to create a bitmap that is sent to the GPU to be drawn on screen. The new method is also executed by Skia, but with a GPU backend called Ganesh. And it's faster because there is no copying involved.

But the challenge of having the GPU rasterise small, complicated shapes like fonts is not trivial at all, especially for the CJK languages with thousands of glyphs. So it seems like both the CPU and GPU will still have their roles to play in the rasterisation process.

If you could have a do-over…

Seedling growing into a flower — What if we actually need a butterfly instead?

Browser rendering engines were designed at a time when GPUs were not commonplace outside gaming machines, and CPUs didn't have as many cores as they do now. Websites also weren't that complicated at the time.

Historically, 2D graphics APIs such as Cairo and Skia have focused on the 2D rendering side of things. But the web platform evolved and we started to do things like animations and perspective transformations in the browser.

To offer first-class support for these capabilities, browsers added them at the compositor level, simply because it was hard or inadequate to do so within cairo or such graphics APIs.

So essentially, browser engines were improved upon based on their existing implementation as computer hardware evolved. Developments such as separating paint and compositing were part of that improvement.

WebRender (1/2)

Drawing of a jet engine labeled with the different Project Quantum projects — The whole web at maximum FPS: How WebRender gets rid of jank by Lin Clark

WebRender (2/2)

Lin Clark's article on MozHacks — https://hacks.mozilla.org/2017/10/the-whole-web-at-maximum-fps-how-webrender-gets-rid-of-jank/

Home page of the Mozilla Gfx blog — https://mozillagfx.wordpress.com/

Acknowledgements

🙏 Big thank you to these beautiful human beings who answered my noob questions 🙏

István “Flaki” Szmozsánszky — @slsoftworks

References

Davis, W. (1986). The Origins of Image Making. Current Anthropology, 27(3), 193-215. doi:10.1086/203422
The GIF Is Dead. Long Live the GIF.
Types of Bitmaps
Why do we need JPG compression and how it's technically working? by Steven Hansen
Progressive JPEGs and green Martians by Jon Sneyers
Finally understanding JPG by Christoph Erdmann
How JPG Works, How PNG Works, Reducing PNG file Size by Colt McAnlis
Thoughts on a GIF-replacement file format
Quantum Up Close: What is a browser engine? by Matt "Potch" Claypotch
Let's build a browser engine! by Matt Brubeck
On rendering engines and graphic libraries by Kilian Valkhof
Following up on the 2d graphics in Rust discussion by Nicolas Silva
Introduction to WebRender – Part 1 – Browsers today by Nicolas Silva
The whole web at maximum FPS: How WebRender gets rid of jank by Lin Clark
Software vs. GPU rasterization in Chromium* by Martina Kollarova
GPU Accelerated Compositing in Chrome by Tom Wiltzius, Vangelis Kokkevis & the Chrome Graphics team

This talk is dedicated to browser engineers everywhere.

I owe my career to you.

I love the web. Truly. And I know that it takes many different groups of people contributing in their own way, that makes the web what it is. I was fortunate enough to attend View Source a couple days ago, and got the chance to meet some of the folks who actually work on browsers.

I know this started out as a talk about images, but as rabbit holes go, you never know where you will end up, right? At this point, I want to show my appreciation for browser engineers. Some of them have made this their full time careers, others have done so as OSS contributors on their own time.

Even though browsers are not the only way to access information on the web, they are the predominant medium at the moment. So thank you, browser engineers everywhere, for striving to make the web experience smoother, faster and more accessible for all of us.

Thank you!

https://www.chenhuijing.com

@hj_chen

@huijing

Font used is Proza Libre by Bureau Roffa

A deep dive into imageson the web

and then some…

Electronic signals

Raster scanning

File formats

Colour depth

MacPaint

MS Paintbrush

Bitmap (BMP)

Graphics Interchange Format (GIF)

Compression algorithms

Joint Photographic Experts Group (JPEG)

JPG compression

Progressive JPGs

JPG optimisation tips

2. Alignment on the 8x8 pixel grid

3. Reduce contrast and saturation

4. Sepia images

5. Slight blurring

Speed, Quality, Size

JPG encoders, there are many

libjpeg-turbo

GIF89a

Burn All GIFs day

Portable Network Graphics (PNG)

PNG filter algorithms

DEFLATE compression algorithm

What a difference 2 pixels can make

Optimising PNG files

3. Use indexed images, if possible

4. Optimise fully transparent pixels

libpng

What is a browser engine? 🤔

What is a browser engine? 🤓

Image encoders in browsers

Browser rendering pipeline

Browser rendering pipeline

Browser rendering pipeline

Browser rendering pipeline

Rasterisation (1/3)

Rasterisation (2/3)

Rasterisation (3/3)

Graphics libraries

Painting

Compositing (1/2)

Compositing (2/2)

Making use of the GPU (1/2)

Making use of the GPU (2/2)

GPU rasterisation in Chromium

If you could have a do-over…

WebRender (1/2)

WebRender (2/2)

Acknowledgements

References

Thank you!

A deep dive into images
on the web