People in Ukraine didn't believe it was going to happen," Dmytro Iolkin recalled. "Not even large companies had backup plans in place when Russia invaded."
As a tech lead at Ukrainian managed service provider Wise Infotec, Iolkin was faced with ensuring his company maintained critical services in the middle of a war.
This was not just about keeping customers online, this was about vital applications, including a security checkpoint app, government workloads, and hospitals. All were running in a data center in Kyiv, which at the time many thought would quickly fall.
"We didn't have a place to go," Iolkin said.
To pull off a rapid data migration in a conflict zone, Iolkin relied on a network of IT technicians and data center workers around the world to keep Wise online.
Among those crucial in making this possible was Zac Smith of Equinix Metal, the colocation giant’s bare metal service. "I knew him from when he was CEO at Packet," which Equinix bought in 2020 to form Metal, Iolkin said.
This feature appeared on the cover of the latest issue of the DCD Magazine. Read it for free today.
He didn't know Smith well, having just spoken to him for a small project that never passed the drawing board. Still, it was the best shot he had of finding some help in a crisis.
"I knew he wasn't on the colocation side of things, but he was the only point of contact I had, so I asked him about that. We first thought about physically moving our hardware," Iolkin said, despite the great difficulty of moving a large number of servers in a war, and the fact that military-age males weren't (and still aren't) allowed to leave Ukraine.
"When all of this started, it was hard to think through things straight - I was just doing what I could," Iolkin admitted. "It was just thinking about how to get to the next day, not how we would actually do it. I had no plan."
Zac Smith had a better idea. "He said 'why don't you just use our Metal service for free?'"
"Wise is hosting critical infrastructure," Smith told DCD. "We wanted to help."
The company had already decided to offer free services to existing Ukrainian customers that were struggling to continue in the face of Russian assault, but was now finding that new companies were reaching out for assistance.
The company took an ad hoc, case by case for new requests. Some received full support for free, others were given huge discounts, where it “is very similar to what we would do with any new customer, except with more generosity,” Smith said. "Like if some things that we have are readily available on our balance sheet and are easier for us to provide, then we can be more liberal.”
The company is still trying to work out how widespread such discounts and free services can be, especially in the face of a lengthening conflict. "To be transparent, it's very expensive for us to provide this infrastructure," Smith said. "It's not like we're providing a $2,000 a month colo cabinet, we're providing a couple $100,000 a month of physical infrastructure."
For Wise, the intervention was a critical lifeline, and one that came within days of the invasion. But companies usually spend months or years planning moves, and it wasn’t an immediate easy fit.
“In Metal, if you have a server that is running and it stopped working for any reason, the assumption is that you would just pick up another server, spin it up and start using it,” Iolkin explained. “But we would have had data on the server, and if it dies the data will die with it.”
The company realized it needed dedicated or distributed storage - either servers just for storage, or redundant storage across the existing servers. “But we figured out that there would not be enough space on the hard drives in the servers that Metal provided us,” Iolkin said. “They are not providing all of the possible Metal options, just those from their 'On Demand' service, and those don't have enough space.”
He told Metal about the problem, and they called in their partner Pure Storage, which installs dedicated persistent storage servers near the Metal servers - and this time offered to do so for free, using equipment from their demo pool. But Pure’s product was in the US, and Wise was moving to an Equinix facility in Frankfurt.
“I didn’t know how long it would take,” he said, remembering the urgency of the situation. “Things can happen during transportation: They could break the system, they could lose it, it could get stuck somewhere for who knows how long.”
Again, he turned to the community for help.
Iolkin had long used hybrid cloud data services and data management software NetApp, and was part of a group known as the 'NetApp A-Team,' a group of customers that evangelize the product and help each other with tricky problems.
He knew NetApp would work as it was what Wise used for its storage system in Kyiv, and he trusted its data transfer and compression capabilities. He sought the A-Team’s help: "My idea was to install the NetApp system on the Frankfurt side and then replicate some or all the data, and then gradually migrate from NetApp to Pure, whenever it would be available."
André Unterberg, a member of the team, put Iolkin in touch with a company selling used NetApp systems in Germany, Miller Anlagen. When he told them about the problem, they gave the systems to Wise for free, for as long as they needed to use them.
"They provided hardware and they delivered the system, and André, Alex Scholz [both from Bechtle], and some other engineers actually installed the system." The whole process took under three days - and then the Pure Storage systems came around a week later.
The data migration happened in stages, first with most of the colder data replicated in Frankfurt, before moving to the live data. It only required a few minutes of downtime "because we did this granularly, virtual machine by virtual machine," Iolkin said.
Wise's network engineer also had to reconfigure the network to a new data center architecture, all as quickly as possible.
In total, 182 terabytes was shifted. It wasn't easy.
"At the time when we were moving all of this data, Kyiv was under attack," Iolkin said. "And when the Russians were bombing, they broke one of the three cables that we used for the Internet connection, so traffic was slower."
The cable was eventually fixed, as part of an ongoing momentous effort by Ukraine's network engineers to keep the country online amid targeted attacks by Russian forces. Their work has helped the world see the atrocities being committed, and has allowed Ukrainians to communicate and plan escapes amid the unfolding horror.
Among those using such connectivity was Iolkin's own family, who had been slow to leave the Kherson region, wishing to stay with their community and in the house they built and lived in all their life.
He didn't hear from them after Russian troops took over the region, nor in the weeks that followed. And then, at last, thanks to repeated fixes to telecommunications networks, he heard from them: "They wanted to move out but it was not feasible," he said, his voice cracking with the memory. "But one week ago, they found a way - and they have moved to my apartment in Kyiv."
For Iolkin, who had been fortunate enough to move to the US for work before the conflict, watching his homeland be invaded, his family go dark, and his childhood home be taken over, was an intensely fraught experience.
"I couldn't sleep at all. I dedicated my time to data migration. Because that's what I could do. That's the only thing I'm a professional at. What else could I do? It was difficult."
While it served as a distraction from the horror, it also had tangible benefits - supporting online tools that are currently being used in Ukraine. It also will help with the nation’s rebuilding.
At the outset of the conflict, he reached out to his alma matter, Severodonetsk University, to move workloads to Wise's data center, and then Frankfurt.
"I wasn't able to reach them in time to get all of it, but they moved some of the data over, so now we have the university running on Metal," Iolkin said. "Now, there is no Severodonetsk."
After a lengthy battle, Ukrainian forces were pushed out of the eastern city. Russian forces shelled the city every day from late February until the end of June, reducing it to ruins.
Iolkin was also able to migrate most of the data out of Kherson State University, which is under Russian occupation.
He, like many Ukrainians, hopes that the conflict will be over soon - but is realistic about the likely long and brutal road ahead. When peace does return, and should the territory remain independent, he is optimistic about rebuilding.
The tech sector could steer clear of Ukraine, or, he argues, it could be crucial in helping the country return to prosperity, showing that Russia does not have the power to destroy a civilization. "We can look at South Korea, which is still in a conflict with North Korea," he said. "Or Israel, where there are a lot of stability issues, and they have some of the biggest tech companies and startups."
He added: "It can go both ways. And that depends on us, on everyone in the world to move it one way and not the other."
That future is still tragically a long way off. We shouldn't wait until then, and allow the continuing horrors to slowly slide down the news pages and into obscurity, lost among the noise of global suffering, as well a local news and the latest frivolities, Iolkin implored.
"You can see it going down and down and down," he said. "People are losing interest in it, and we need to speak up, we need to remind what's going on," noting that such vigilance was crucial not just for the people of Ukraine, but to prevent Putin expanding further, nuclear weapons proliferation, and China being emboldened to invade Taiwan.
Throughout the conflict, Wise has continued to operate its data center in Kyiv - with some customers unwilling, or legally unable, to leave the country. Staff continue to work there, moving in and out to the safer western edge of the country.
Shelling continues in Kyiv. "Fortunately, they never hit the data center in Ukraine. They hit lots of places, but not the data center - it hasn't gone down once,” he said proudly.