Why Brotli Adoption is so Low Among Websites | Dexecure

Why Brotli

Brotli is able to consistently achieve better compression than Gzip. Studies conducted on real-world web assets show a 39% better HTML compression, 26% better JavaScript compression and 17% better CSS compression over Gzip at the 90th percentile. Brotli is also great at compressing newer data types like WebAsssembly.

That cool AngryBots WebAssembly demo?

AngryBots13.wasm
------------
Served: 10.56mb
gzip: 3.6mb
brotli: 2.4mb

Brotli all the things!

— Eric Lawrence (@ericlaw) November 11, 2016

Support status

Brotli is mostly supported only on HTTPS websites (for good reason) by Firefox and Chrome. Other Chromium based browsers like Opera, Brave and Vivaldi support it too. That is quite a lot of people who can benefit from Brotli right now! Support for Edge is in development. Bug (no pun intended :P) Safari to add support too.

So if considerable number of people are on Brotli-enabled browsers and it is obviously better at compression than Gzip, you would except a lot of websites to be using it. Sadly, you would be wrong. A quick search on BigQuery shows that only 357 domains are currently serving Brotli.

So why aren’t more websites using Brotli already if it is so awesome?

Huge websites like Facebook and Google are using it in production. The main reason for the slow adoption rate among other websites is because it is not supported by a lot of the major CDN providers. CDN providers like CloudFront, Cloudflare and Max CDN normalise the Accept-Encoding header to just gzip before making the request to your origin server. Therefore you are unable to detect if the client actually supports Brotli and send a different asset to them.

Out of these 357 domains, these are the number of domains serving Brotli via a CDN. The only commodity CDN supporting Brotli seems to be CDN77 and there are some hacks to support Brotli in Fastly too.

If you are like me, you cannot wait for other CDN vendors to enable Brotli support. This is possible using some Service Worker awesomeness.

I wrote a small Service worker script to get around this. Service workers are supported in all browsers that support Brotli - so you don’t need to worry about Service worker support (are we seeing a pattern here among browsers? hmm…)

The idea is to find if the client supports Brotli and attach that information as a header (or a query string) with each request so that the server can respond differently. The service worker is a great place to do such transformations to requests since it is able to intercept any outgoing request from the browser.

However, it is not trivial to find out if the clients supports Brotli from within the service worker environment. Usually, this information is broadcasted within the Accept-Encoding header. This header is a protected header as defined by the Fetch Specification and is not accessible from within the service worker. Although, you could do some sort of UA parsing or assume all browsers that support Service Workers support Brotli, it is not very future proof. (For example, Edge could land their service worker support before the land their support for Brotli)

To get around this, we download a Brotli encoded file from a URL in the install event. In the gist above, I have encoded a simple text file with the word Dexecure and hosted it on Cloudflare. If the browser supports Brotli, the response would be correctly decoded in the res.text() call. If not, it would just be junk.

Based on this, we set a variable in the service worker indicating if Brotli is supported or not and attach an header (or a query string) with every request and your server can distinguish and handle these clients appropriately. (Make sure you whitelist or vary by that header in your CDN!) However, maintaining global state in service workers is not a good idea since they are terminated by the browser after events (like fetch, push) are handled. To persist information, we store the information in IndexedDB.

Here is the code for the same. If you want the code that saves the info to indexedDB too, check out this gist.

This is the method we use at Dexecure too to enable support Brotli encoding among a larger set of users. Hope to see that number of domains using Brotli go up in the next HTTP Archive crawl 😃