Launching a new online video game is hard – besides making something that’s actually enjoyable to play, you have to think about latency and bandwidth, doing everything in your power to avoid lag – the delay between player input and action on the screen.

Lag is born when networks and servers can’t keep up with demand – when developers don’t reserve enough capacity, can’t scale fast enough, or their data centers are located too far away.

This is less of a problem than it used to be, but AAA game launches are an exception: player demand is hard to predict on day one, and a simultaneous worldwide release of a hotly anticipated title can, in effect, resemble a DDoS attack. Such launches rarely go smoothly [AAA is gamer-speak for "major" - Editor].

For this reason, the recent arrival of Respawn Entertainment’s Apex Legends was such a surprise – the game attracted 10 million players in 72 hours, and saw at least one million on its servers at the same time. Despite this unprecedented level of demand, the uptime remained (almost) rock-solid. Two weeks later, Apex is on track to become the most popular game in the world.

What is its secret, you ask? Respawn went with public cloud – and to improve resilience and geographic reach, not just one, but the world’s three largest public cloud providers: Amazon, Google and Microsoft.

Uptime legends

The struggle against lag is an important part of gaming history.

“The PlayStation Network during the PS3 era was a terrible service, designed to test the limits of gamers' ability to handle error codes, unscheduled maintenance and loading screens,” said our own Seb Moss, who was reporting on the subject at the time.

“The PS3 was Sony's first real 'online-ready' console, except Sony was not ready to go online. Slowly and sporadically it improved - the company struggled to invest in the free service as its gaming division hemorrhaged money.

“These were not good times for Sony, nor for players. It was a great time to be a games journalist covering downtime, though.”

At the same time, game developers that treat their infrastructure with the respect and attention it deserves pass into legend – like CCP and its Eve Online, a persistent, incredibly detailed online world that has been going since 2003 and still enjoys, on average, more than 20,000 players at any one time.

Eve is famous for its network architecture, which enabled some of the largest virtual conflicts ever - like the infamous Bloodbath of B-R5RB in 2014 that saw thousands of players contest one of its 7,800 star systems for 21 hours straight.

Apex Legends does things on a much smaller scale – it features 60 people per single game world, each with their unique combination of weapons and equipment. The game exploits the popular Battle Royale formula, seen in games like Fortnite and Player Unknown’s Battlegrounds – one round takes around 20 minutes, during which players hunt each other until only one team remains.

Unlike Eve Online, a game about starships that is lovingly referred to as ‘spreadsheet simulator’ by its fans, Apex offers something called twitch gameplay - a reaction-based process where a fraction of a second can mean the difference between victory and defeat. In this scenario, responsiveness is paramount, and any technical issues will see players leave.

The game wears its infrastructure credentials on its sleeve: as you log in, a tab called ‘Data Centers’ is one of the first things you will see. It shows a total of 44 different facilities around the world: Google Compute Engine sites are easy to identify by the ‘GCE’ tag – the rest are a combination of facilities from AWS and Microsoft Azure, plus some bare metal servers.

With public cloud infrastructure, scaling up is easy – just add more virtual machines. Another one of Respawn’s achievements is non-disruptive updates. In online gaming, it is commonplace to schedule long maintenance windows, during which servers are inaccessible and updates can be applied. Thanks to its flexible infrastructure, Respawn has been rolling out updates without having to stop the game.

“This has been a truly incredible journey. We tested and tweaked. We argued and agreed. We got to a point where we felt some magic,” Respawn CEO Vince Zamplella said in a blog post on the game’s sudden popularity.

The same company was previously responsible for the hugely successful Titanfall series – another game with a strong online component – and that’s where it learned to love the cloud.

Engadget reported that the original Titanfall relied on infrastructure from Microsoft. The sequel was a bigger, much more complex game – and the creators opted for a multi-cloud architecture, managed using software from UK-based Multiplay. Back in 2016, this was an act of considerable bravery.

"Since this launch is going to be even bigger than the last game, I really wanted to make sure that we had an insane amount of scalability and reliability," lead engineer Jon Shiring said at the time.

More recently, he tweeted that Apex Legends’ network model and netcode was developed by a team of just three people. If this doesn’t suggest the maturity of the multi-cloud model, I don’t know what does.

Cloud computing is already enabling a quiet revolution in gaming, but having a single cloud provider is not enough – just think about Pokemon Go, which relied exclusively on Google Cloud in 2016, and paid the price; developer Niantic was forced to pause the global launch, while it was beefing up its infrastructure.

Apex Legends’ publisher Electronic Arts would be wise to look at the multi-cloud approach for its recently announced game streaming service, Project Atlas, as well as any other upcoming AAA titles.

A pipe dream just a few years ago, today game streaming is being worked on by dozens of teams - from startups like Blade to Internet juggernauts like Amazon. Google is widely expected to unveil its own game streaming tech at the Game Developers Conference in San Francisco in March.

Gaming-as-a-Service is the Holy Grail of digital entertainment, enabling customers to play the latest titles without the need to invest into expensive hardware. What I have experienced playing Apex Legends makes me think that our infrastructure is finally ready for this.