It’s easy to forget how each proceeding generation of technology improves on the last. Often, those improvements are evolutionary, not revolutionary, but over a period of just a few years, they make a big difference.

When it comes to the technologies that keep data centers running, such as UPS and backup systems, the replacement cycles can be counted in decades rather than years. The technology leaps that data center operators make when they upgrade are so monumental due to innovation that costs can be slashed, space freed-up and processes radically improved, all at the same time.

However, upgrading UPS systems, back-up generation and associated electrical systems is not a straightforward task. Especially when the data center is located in an older building in the middle of a busy city, and the generators are sited several floors up.

But that is just the kind of project that power specialist Hitec and professional services program management firm GRA were engaged for recently. The project not only involved upgrades to ten diesel rotary UPS (DRUPS) systems, but also necessary related electrical systems. A total capacity of 25MW was upgraded.

Furthermore, the technology leap was such that the organization was also able to benefit from a major update to its business processes, helping to radically ratchet-up the data center’s efficiency. Indeed, the installation of the new units reduced operational costs by 25 percent, while improving operational efficiency, reliability and reducing annual maintenance.

Hitec’s DRUPS technology combines energy storage and energy transfer in one unit, while also providing power conditioning. The units include a kinetic energy module, a generator, and a diesel engine, which is a connected but standalone element of the overall system. In the event of a power outage, the kinetic energy module is able to provide instant energy in the event of a power outage, while the diesel engine provides the longer-term backup power, if and when required. The system includes all controls that make it work as one.

“We’ve been retrofitting all-new electrical components, all-new electrical infrastructure,” Robert Suchon, business development director at Hitec Power Protection, tells DCD. “It was a total refurbishment of the main electrical infrastructure, the DRUPS – diesel rotary UPS – which uses electro-mechanical flywheel technology to store energy, as against using multiple batteries.

That meant “all electrical infrastructure, and control systems, and the complete change-out of the DRUPS,” says Suchon. Not only that, but it was also the first retrofit in the field for this particular product line, Hitec’s PowerPro 2700. He adds, though, that the electrical distribution technology remained untouched, and the configuration and sequence of operations remained the same. “All we did was modify it to ‘talk’ to the new Hitec control system.”

However, this was not a casual swap out. The units were so hefty they had to be craned in. “It all had to be done on the roof mezzanine of a 100-year-old downtown, San Francisco building. We had to shut off some streets over several weekends during the Covid-19 pandemic, and we were also doing some of it, quite literally, during the social unrest last year.”

The systems were around 20 years old, but coming up for their next major five-yearly overhaul. So rather than the standard refresh to cover the next five years, the client decided on a major upgrade. Some of the benefits of such an investment included the ability to take advantage of new features, and improved reliability.

Furthermore, the units will now provide a further 20 years of reliable service. “And the new generators are much more efficient and more reliable, while the maintenance costs have been halved. They also sit on a smaller footprint, so there’s some space savings as well.”

The new equipment also offers a number of features that can help improve efficiency.

“We now have an integrated automatic greasing system, for example, which injects grease every few minutes. So, when the unit is operating, it automatically does the lubrication that it needs on its own. Staff can stay remote and don’t have to go on-site with a grease gun to inject lubricant into the bearings.

“In addition, the way the technology has been designed means that it runs at a much slower RPM and a lower temperature, which translates to less stress on the technology. This design doubles the life expectancy. Typically, such units had preventative maintenance intervals of around five years; now, it’s after ten years. However, what we’re expecting is that it will be closer to 15 years,” adds Suchon.

This increased lifespan is achieved by a new kinetic energy module design, he continued, enabling the speed of the flywheel to be significantly reduced and controlled, meaning less stress and a lower operating temperature on the bearings.

“So, the automated lubrication with the reduced speed has enabled us to extend the overhaul periods and cut the maintenance window by 50 percent. The older technology required the bearings to be changed at a five-yearly interval. With the new technology they only need to be serviced after 10 years of operation.

“The next feature designed into our system is integrated real-time vibration monitoring across the entire power generation module. At any point, you can see that it's running in a good, steady state of operation. Users get significant monitoring capabilities with this new technology versus what was available 20 years ago, when our initial controls were installed,” says Suchon.

The upgraded technology alone was able to save the client 250 kilowatts of power, amounting to hundreds of thousands of dollars in annual savings, says Suchon, as well as helping to free-up staff from some routine maintenance.

Partners in grime

While Hitec was the technology provider, with a number of other companies providing electrical and mechanical infrastructure – not to mention the hard and sometimes grubby work of installation – GRA was brought in as the program manager and data center owner’s representative. GRA’s role included assessing and validating the operating assumptions that the Hitec team had put together in the design phase, including maintenance, operating efficiency, performance, improvement of uptime reliability, as well as the overall installation plan.

However, the partnership developed into something much, much more. “We were also asked to look at the implementation and document it in preparation for the construction phase, because the installation was going to take place in an active operating environment,” says GRA president Joseph Ryan.

Initially, the client wanted a no-nonsense, traditional implementation methodology.

“They allotted four weeks per ‘unit’ [of work] in their budgets and schedules and managed it themselves in the way they traditionally handled such a project. But the first unit took 12 weeks,” says Ryan.

“So, we started to inject new tools and technologies into the process to push it along at a faster pace.”

GRA’s approach is adapted from Lean – The Toyota Way – and the Last Planner^® System of production control. Furthermore, the client was also able to learn from each unit installation and apply the lessons to the next: to learn, refresh and reset. “We started to implement the tools and methodologies that we have at GRA.

“Showing the client by means of lessons learned, we were slowly able to show the value of working ‘The GRA Way’. They had to try it their way first. As obstacles and delays kept the project from staying on schedule, it opened the door for us to show them a better way. You’re working with massive infrastructure requirements. The only way to be truly successful is to document it and improve it as you proceed,” says Ryan.

To help ensure the schedule for the next installation, the owner allowed for the introduction of Lean principles. This created a space for deeper conversations around scheduling. Planning sessions, coupled with daily status updates, set the tone for improved collaboration between the owner and the subcontractors. Typical planning tools were used to manage the work: Excel spreadsheets, emails, Word documents, and several virtual and in-person meetings, and so on. Although this effort accelerated the schedule, the second installation still took seven or eight weeks to do.

At this point, GRA went a step further. “We began to influence the traditional culture of the organization. That’s partly why we were brought in – to help transform for the better.

“Technology and tools can’t solve the whole problem; people alone can’t solve the problem. It’s the combination and balance of the sum of all these capabilities, along with the creation of an integrated approach that solves the problem.

“By the third unit we were at four weeks of installation, trending down to three weeks, so we were now gaining efficiencies along the way, as well as improving on the original plan,” says Ryan. Ultimately, he says, it was all about driving through as much standardization as possible.

Some of the tools adopted included VisiLean for production planning, Autodesk BIM 360 for information and field management, and even virtual reality was used as a collaboration tool with Facebook Oculus.

The project also required pulling together teams in California, Texas, the US East Coast, and the Netherlands, where the products are manufactured, collaborating throughout the process of design, manufacturing, construction, and installation.

That was easier said than done with measures to contain the spread of Covid-19 coming, going, and then being re-imposed, not just in different parts of the world, but also in different parts of the US. GRA introduced 3D modelling, communications, and scheduling tools to help do that and to maintain ‘business as usual’, as far as possible.

In the process, the one-hour or more project meetings, four times a week, at the start were cut to just one weekly meeting of around 20 minutes, leaving more time for the real work to be done. This effectively reduced staff workload by four hours a week for 10 people, or 40 hours across the week for the entire team.

“It really boils down to changing the culture and changing the way people approach work, and breaking down old methodologies, persuading people not just to come into work and do their own stuff, but genuinely collaborate,” says Ryan.

After all, he adds, the infrastructure upgrade carried with it major risks to the client and continuity of service that had to be closely managed and intelligently contained.

In the process, the perception of GRA as simply the project manager appointed to make sure the client company’s interests were looked after shifted to one of bringing new ideas to the organization, and helping kick-start a much-needed culture modernization.

Indeed, says Ryan, too often organizations embark on potentially revolutionary multi-million or even multi-billion-dollar investments, while clinging to old methodologies and ways of working, blunting the potential impact of the project.

However, by gathering data, learning, iterating and improving, the project has lasted only about a year, from start to finish, when before GRA’s appointment it was on course to take at least two years, says Ryan.

Getting it right

It wasn’t necessarily straightforward, at least in the beginning, says Suchon. Hitec was responsible for the technical design aspect of the project, including mechanical and electrical integration.

“We needed to work very closely with the installing contractors and GRA to ensure all the scopes of work came together seamlessly, from removing the older infrastructure, taking it out of the building, bringing in the new technology and making sure everything fitted together. Then there were all the wiring and mechanical pieces, spinning up the units, testing and putting them back into service,” says Suchon.

He adds: “It was not a large project, but it was technically very complex. The owner set out a plan and then pretty much told GRA how to do it: traditional methods, typical schedule to follow, and it didn’t work. So, here, we pushed to try some new tools and methodologies.

“Then, Covid came in and we couldn’t continue to work the way we used to, anyway. The client adopted some of these new proposals, although there was some resistance because people had been doing it the same way for a long time.”

However, a series of apparent improvements feeding through from the second set of installations helped bring people onside. “With that momentum, we were able to go step-by-step and, when it came to the final installation, we got there one week early: collaboration meetings that were taking two-and-a-half hours several times a week are now done in 20-to-30 minutes, and some are virtual. That has helped streamline the whole process,” says Suchon.

The clients, meanwhile, have enjoyed a swift upgrading of attitudes, with Hitec and GRA bringing good new ideas to the table, bringing demonstrable benefits during the installation that can be continued for the organization’s benefit long after both the project and Covid-19 have become a distant memory.

“This was a live, functioning data center throughout the entire project, so the coordination between us, GRA and the various contractors – not to mention with the client – was absolutely critical. We weren’t afforded any mistakes: the technology had to work once it was put back into service,” concludes Suchon.

This project was covered in detail during a presentation at the recent DCD>Critical Power broadcast, which is now available to view on-demand.