Facebook has built its own network hardware to speed up the traffic between data centers within a region. The Fabric Aggregator, shown at the Open Compute Project Summit in San Jose, is based on multiple switches - specifically the Wedge 100 switch which Facebook already donated to the OCP effort.

As Facebook expands its data centers - it recently announced its Papillion facility would grow to six buildings - the amount of traffic flowing on its network is growing massively, product manager Sree Sankar told the OCP Summit. Facebook has therefore built its own distributed network system, with features aimed at accelerating “local” traffic - i.e. data traveling within one region.

Local traffic dominates

Facebook’s network traffic is distinguished between the traffic within its local region (or “East-West” traffic), and that which has to go upstream to other regions (“North-South”). The company’s network solution is based on an aggregation layer which handles these two types of trafrfic differently. 

”Separating the solution into two distinct layers allows us to grow the east/west and north/south capacities independently by adding more subswitches as traffic demands change,” Sankar explained in a blog post.

Instead of general-purpose switches, Facebook is using specialised aggregation nodes built from its Wedge 100 switches, which the company designed internally, and which were accepted by the OCP as open source hardware in 2016. These run the Facebook Open Switching System (FBOSS) software, also contributed to OCP.

Ech node is a two-layer cross connect, and the capacity of the network can be increased by simply increasing the number of nodes in the aggregation layer. 

The total size of a Fabric Aggregator node can be quite large, Sankar said: ”The physical footprint of this unit of bandwidth in terms of space, power, and cooling, may or may not fit in a rack. Early into our design, we recognized that the most flexible designs would need to be larger than a single rack.”

The architecture allows single-rack or multi-rack aggregators which can be chosen based on space, power and other constraints, she said.