Impact of 3G vs 4G Connections on User Experience across Countries

It is no secret that the faster the user’s internet connection is, the faster the website will load. It is a crucial factor in the success of a business, but one that it can not directly control (long live #netneutrality). It is also hard to quantify how connection speeds directly affect user experience, and consequently important business metrics such as bounce rate. In this post, we use Chrome User Experience Report to analyze the impact of internet speed on real-world user experience. To know more about the data and its use cases, see our introductory article - Chrome User Experience Report Explained with Google BigQuery.

How a Website is Affected by Connection Speed

It has been well established that the page load time is not a good metric to capture the loading experience of websites. Metrics such as Time to First Interactive (TFI), First Meaningful Paint (FMP) and First Contentful Paint (FCP) provide a much better insight into the experience of the user as he is loading your website. We will focus on the Time to First Contentful Paint (TFCP) - the time at which some content (text, image, background, etc.) is painted on to the user’s screen. This question can be asked in a variety of contexts (and we will in subsequent articles, follow us on twitter for updates!), but this article focuses on its relation with connection speed.

There are five effective connection types in the report: 4G, 3G, 2G, slow-2G, and offline, which is in accordance with the effectiveType attribute of the Network Information API - recently launched in Chrome. Note that this API is not restricted to mobile devices and the Chrome UX report has this data collected for all device types.

Recall from our previous article that we are focusing on Site Experience Benchmark (SEB) - the probability of TFCP being less than 1 second. Given a website, we find out how SEB changes over different connection speeds. What is SEB for a website on a 3G connection? What about 4G? The probability will very likely increase, but how much? What if you define SEB as probability of TFCP being less than 2 seconds instead of 1 second? How much better would the results be?

We will use BigQuery to answer such questions (see here for a quick primer). The following query returns the density of data points for the FCP metric loaded with a 3G connection for https://www.google.co.in. The value is 0.2583.

Similarly, the following query returns 0.1876, the total density for the TFCP metric being less than 1 second for https://www.google.co.in in 3G:

Hence, the probability of https://www.google.co.in with TFCP < 1s in a 3G connection is 0.73 (= 0.1876 / 0.2583). Thus we conclude SEB=0.73 for https://www.google.co.in in 3G. However, if we ask the same question for a 4G connection instead of 3G, SEB increases to 0.84 (= 0.5968 / 0.7147), a 15% increase (= (0.84 - 0.73) * 100 / 0.73).

By moving from 3G to 4G connection, the SEB (Site Experience Benchmark) for google.co.in improves by 15%.

Based on your use case, you can choose a different value instead of 1 second when defining SEB. The following visualization shows how SEB improves over time for https://www.google.co.in.

Caution: It is tricky to do this analysis across origins and come up with meaningful results. For example, we can not compare the improvement of SEB by changing connection type for two different websites. The improvement can differ for a variety of reasons, including the size and complexity of the website.

State of 3G and 4G across 72 Countries

While we have derived how an individual website is affected by the change in connection types, it is also debatable if connection speeds remain the same across the countries – is 3G in the US the same as 3G in India? A number of global internet speed comparisons studies exist. Some of them are synthesized experiments, while others focus on ping response times and upload/download speeds. Here, we will focus on how SEB is affected across the world with the same connection type.

While the Chrome UX report gives us real-life data for Google Chrome users, it does not provide any information on the location of the user (yet). Hence, we can not compare the speed of an arbitrary website in different countries, nor can we compare two websites from different countries. However, if we know that a website has most of its traffic from a particular country, we can fairly assume that most of the data collected on the website is from that country as well. If and when Google includes location specific data into this report, this analysis will be updated accordingly.

We wanted to find websites that have country-specific versions. There are a number of such websites, but most of them, quite understandably, have varied style and content across countries. Finally, we settled on google.com which has data points for its 75 country specific websites in the report. We cross checked with Similarweb and found out that .com, .ru, and .com.hk versions of Google have significantly lesser traffic from US, Russia and Hong Kong respectively. So, we include the remaining 72 country specific Google domains that now has more than 90% traffic (according to similarweb) from their respective countries. Now, we assume that the data from a country-specific version of Google (say, https://google.co.in) represents the state of the connection in the country (say, India).

We now see the SEB for country-specific Google domains for all the 72 countries. When restricted under connection speed, this shades light on the state of the connection across the globe.

State of 3G across Countries

#My3GisNotYour3G: SEB for Google in 3G is 30% worse in Venezuela than Sweden.

State of 4G across Countries

#My4GisNotYour4G: SEB for Google in 4G is 25% worse in Nigeria than Sweden.

Our analysis showed results consistent to overall state of internet connectivity report across regions.

Key Takeaways

• The latency and bandwidth of the internet connection plays a very important role for a website.

• Frustratingly enough, the business can not control it.

• However, being aware of this quantitatively can help you take better decisions as you design and architect your website.

• Using RUM data from these websites show how different connection speeds across the world has a very direct impact on your performance metrics.

• Synthetic monitoring should always be coupled with RUM monitoring to get a complete picture, even simulating a 3G connection is not enough.

• What we define as 3G (or 4G) varies from country to country. Use a real device and test on real networks based on your target user’s profile.

• Monitoring tools like Calibre and Speedcurve can help understand your user’s environment better, so that you can take steps to better optimize for their network conditions.

Read The Other articles on Chrome User Experience Report

In coming weeks, we will analyze the report for a number of new questions in separate blog posts. We will update the following list as we go along, so do not forget to subscribe to our blog!

• Chrome User Experience Report Explained with Google BigQuery.
• [Impact of a Country Specific Website on User Experience] (/blog/impact-country-specific-website-user-experience/).
• What is the impact of Server Side Rendering (coming soon).
• More articles coming soon. Stay updated!