When file hosting service Dropbox first announced its hybrid cloud effort Magic Pocket in 2016, many saw it as a sign that the company was done with Amazon Web Services and was betting on an on-premise future.

But the reality is more nuanced, lead developer Preslav Le told DCD.

The company has always had its own data center presence, but Dropbox needed more capacity and soon grew to become a major customer of Amazon S3 (Amazon Simple Storage Service) after joining in 2013. It didn’t take long for the company to wonder whether it made more sense to do it themselves.

"We used AWS S3 because storage at scale was an extremely hard problem to solve," Le said. "It was only a few years later, when we really believed we could tackle this problem better for our needs, that we even tried."

The result was Magic Pocket, one of the largest data migrations off the cloud in web history. This, Le said, has allowed for significant cost savings and more control - but is not something that most other companies could easily replicate.

This article appeared in Issue 42 of the DCD>Magazine. Subscribe for free today

Over a two-and-a-half-year period, the company built its own massive on-premises platform, officially launching it in 2015. This involved a huge amount of software work - including switching from programming language Go to Rust mid-way through to reduce memory use - and getting deeply involved with the hardware to ensure that every ounce of possible storage was squeezed out of a rack.

"It's not only the language we changed," Le said. "We also significantly improved the architecture. We moved from using a file system to just managing the drive directly - we literally open the drive as a block device and then we have our own formats. This allowed us to gain a lot of efficiencies from avoiding the file system, but also move quite a bit faster."

For example, the company could adopt shingled magnetic recording (SMR) hard disk drives without waiting for drivers to support them. SMD disks can be much denser by writing new tracks that overlap part of the previously written magnetic track, somewhat like overlapping roof shingles.

"This is one of the examples where we were able to work closely with hard drive companies and were able to move much faster than some other companies," Le said. "They need to build a new file system, etc. Some of the big players still don't use SMR."

The company helps design its own custom servers, cramming more and more storage into its data centers. "We replace our hardware every four years, but have at least a couple of new generations in those four years," Le said. "Back when we started, we worked with four terabyte drives. Now we have 20 terabytes... but we also increased the number of drives per chassis so we really increased the density quite a bit."

By 2016, the company said that it had moved around 90 percent of its files over to on-prem, comprising four data centers. "What we've seen in the last couple of years is that we tend to move more things on-prem than towards the cloud for our core storage production," Le said, but declined to share the exact percentage.

The initial move was a big risk. "Looking back, it really turned out to be a great investment for both our velocity and the business," Le said. "Amazon and the cloud have to solve really broad problems - just imagine all the different usage patterns for S3. Our usage patterns are much simpler, and we understand them, so we can [build for them]."

So does this mean Dropbox has dropped the cloud, and is essentially an on-premises business now?

Not so, Le argues. "Magic Pocket is this very famous system, and often people say 'what's the Magic Pocket team?' We don't have one, we have the Storage Team. The reason we call it Storage is because their job is not to do Magic Pocket.

"Their job is to provide the best, most reliable and cost-efficient storage for Dropbox. So if ever Amazon can innovate and they're better than us, and they're cheaper, or we can secure better deals wherever makes sense, their job is to advocate us moving the data back."

Indeed, in places where Dropbox doesn't have the scale, or prices differ, it still relies on S3 - including the UK, mainland Europe, Japan, and much of the non-American world. It does, however, operate its own Point of Presence network.

It’s all about keeping one’s options open, Le said. "For the initial migration out of S3 to Magic Pocket, we built the ability to move data back and forth between the two locations. Over the years, we decided that it's worth retaining that capability.

"So if we ever decide because of supply chain issues, Covid, or whatever, that you want to spin over some capacity to S3, we can just do it with a click of a button - we don't need to write code, you don't need to deploy, you can literally click a button and then some data can go back."

He added: "If adopting other cloud providers made sense, we'd do that too.”

There are other areas where the cloud comes first, too. "Some workloads from our analytics and dev box and other auxiliary things, we've moved to the cloud, where we can allow people to move faster and the cost is acceptable."

The cloud still makes sense for most businesses, Le said. "I think if you're starting a company, just go use the cloud. Operating your own infrastructure comes with a cost.

“And the only way to justify it is if A) You have a very good understanding of the problem. B) You have the right scale - usually, that means a huge scale: with Magic Pocket we store exabytes of data. And then there’s C) Do you have the right talent?"

Dropbox is also fortunate that it is primarily a storage-focused company, so it's hard to get locked into the cloud. Users of more specialized cloud services or databases are increasingly finding themselves trapped on platforms that are hard to extricate their workloads from.

"Sometimes vendor lock-in is ok when building a prototype. It's a small scale, it's not expensive, just go use AWS. But if you're building something where your business margins are seriously affected, then you should seriously think of vendor lock-in."

That's why, if you have the scale and the team, "you should try to really embrace hybrid cloud," he said.

The cost of R&D on Magic Pocket "has not been hard to sustain" since the initial flurry of investment in the shift. "There are all these other costs like hardware and data center operations but whenever we compare costs, we take all those things into account.

"Magic Pocket was a really sound investment that really paid off multiple times over."