Engineers at Facebook have created a custom PCI Express card which serves as a very accurate Time Appliance, and released it as open source, so distributed systems can benefit from microsecond-level synchronization.
Since March 2020, Facebook has been switching its data center servers and consumer products to a timekeeping service based on the Internet's Network Time Protocol (NTP), which increased the accuracy of timekeeping on Facebook's infrastructure 100-fold, from 10 milliseconds to 100 microseconds. The Time Appliance has been built into a PCI Express card, which links to accurate time provided by the Global Navigation Satellite System (GNSS), and which can be installed in any server to provide an accurate time service The design will be shared via the Open Compute Project (OCP).
Time flies
All distributed systems need an accurate time service, and most rely on a hierarchy that goes back to "Stratum 1," usually the GNSS or a cesium clock. Facebook already provides a public time service at time.facebook.com. However, these services are only as good as the Internet connection you have to them, and if the connection to the time system is broken, the local time used by servers can drift, and they will get out of step with other parts of any distributed system.
"To remove these dependencies, we’ve built a new, dedicated piece of hardware called a Time Appliance, which has a GNSS receiver and an atomic clock (often rubidium)," say Facebook engineers Oleg Obleukhov and Ahmad Byagowi in a blog post. "Users of Time Appliances can keep accurate time, even in the event of GNSS connectivity loss. While building our Time Appliance, we also invented a Time Card — a PCIe card that can turn almost any commodity server into a Time Appliance."
Time appliances already exist, but those available on the market are usually based on outdated technology and can be vulnerable to security attacks, according to Obleukhov and Byagowi. Resetting the time on a server can allow some dangerous exploits. The Facebook duo also dislike the use of proprietary closed-source software, which limits the way they can be configured or monitored, and proprietary hardware which is more expensive and cannot be serviced by the user.
The pair prototyped a system using an Intel x86 processor, which takes its signal from a GNSS receiver, which gives one pulse per second (PPS). Backed by a GPS disciplined oven-controlled oscillator (GPSDO), the prototype gives time on the card which is nanosecond-accurate, and can be used to improve accuracy across the network using an off-the-shelf network card, down to "a couple of tens of nanoseconds."
From there, they worked on a PCI Express card version with an onboard miniaturized atomic clock (MAC), a multiband GNSS receiver, and an FPGA to implement the time engine. "The time engine’s job is to interpolate in nanoseconds the granularity required between consecutive PPS signals," the pair explained. The system generates the time of day as well as the pulse-per-second signal. If the GNSS signal is lost, the atomic clock keeps it on time and steady.
"It should be noted that the accuracy of a GNSS receiver is within tens of nanoseconds while the required ongoing synchronization (calibration) of the MAC is within 10 picoseconds (1,000 times more accurate)," the pair point out. "At first, this sounds impossible. However, the GNSS system provides timing based on continuous communication with standard time. This ability allows the GNSS onboard clock to be constantly synchronized with a source of time provided to its constellation, giving it virtually no long-term drifting error. Therefore, the MAC’s calibration is performed via a comparison of a MAC-driven counter versus the GNSS-provided PPS pulse. Taking more time for the comparison allows us to achieve a higher accuracy of calibration for the MAC."
When the GNSS signal is lost, time accuracy deteriorates, but the atomic clock can keep time to within one microsecond for 24 hours. The card will work with any x86 server: "The beauty of having PCIe cards is that the setup can be assembled even on a home PC, as long as it has enough PCIe slots available."
The decision to open-source the card, rather than just use it in Facebook was natural, say Obleukhov and Byagowi: "Building a device that is very precise, inexpensive, and free from vendor-lock was an achievement on its own. But we wanted to have a greater impact on the industry. We wanted to truly set it free and make it open and affordable for everyone from a research scientist to a large cloud data center.
"That’s why we engaged with the Open Compute Project (OCP) to create a brand-new Time Appliance Project (TAP). Under the OCP umbrella, we open-sourced everything at www.opentimeserver.com, including the specs, schematics, mechanics, BOM [bill of materials], and the source code. We also worked with several vendors (such as Orolia) who will be building and selling the Time Card on the open market. Now, as long as printing the PCB and soldering tiny components does not sound scary, anyone can build their own reliable Stratum 1 NTP or PTP time server for a fraction of the cost."