About ExpressFabric | Relevant Videos | Published Articles

ExpressFabric: Thinking Outside the Box

ExpressFabric® is the revolutionary PLX platform that enables, for the first time, a converged fabric to be deployed using the universal PCI Express interconnect. The fabric allows:

The Soul of a New Interconnect

PCI Express (PCIe) has been deployed throughout nearly every market since the first version was introduced in 2003. The specification has progressed from a per-lane data transfer rate of 2 Gb/sec, to the current Gen3 per lane transfer rate of 8 Gb/sec, and can be aggregated to allow a bidirectional data transfer rate as high as 256 Gb/sec.

Until recently, the success of PCIe has been primarily as a fanout interconnect, enabling CPUs, I/O, and storage devices – all of which have a PCIe interface – to communicate. There has been penetration into more sophisticated applications, such as host failover, and the PCIe interconnect standard has even been used as a backplane to connect PCIe-based subsystems. But given the performance of PCIe at Gen3, and its widespread adoption on devices, the popular interconnect has become an attractive alternative to current solutions as a fabric for cloud and enterprise data center applications.

PLX has extended the reach of PCIe for use as a cloud and enterprise data center fabric through its ExpressFabric initiative. By building on the natural strengths of PCIe – it's everywhere, it's fast, it's low power, it's low latency, it's affordable – and by adding some straightforward, standards-compliant extensions that address multi-host and I/O sharing applications, PLX has created a universal interconnect that substantially improves on the status quo.

Saving Space, Money, Power, and Effort

PCIe is the main connection to the world on almost every device used in the data center. CPUs, communication devices, storage devices, GPUs, FPGAs – even other fabrics – connect to each other through PCIe. Using PCIe as the main fabric for the data center rack eliminates the cost, power, and latency of the "bridging" devices that make up the status quo today: primarily Ethernet and InfiniBand.

The backbone of the cloud or enterprise data center will continue to remain mostly Ethernet, with InfiniBand supporting HPC installations, but PCIe will serve as the fabric within the rack. This peaceful co-existence – with each technology matching its technical capability, will be the norm.

Ethernet & InfiniBand, the incumbents, have advantages – Ethernet has low cost and a large software base; InfiniBand has low latency and high performance. But neither has the advantages of PCIe:

Software Defined Fabric

ExpressFabric is built on a foundation that combines dedicated hardware; initialized and managed by firmware.

The fabric is defined by several different types of connection points – or ports. Each port on the fabric can be specified to be one of these, and takes on specific characteristics once it has been defined:

Creating a Synthetic Hierarchy

One of the main obstacles to creating a fabric with PCIe has been attaching multiple hosts to the network without a lot of custom enumeration and software. ExpressFabric completely removes that issue, and does so with standard PCIe enumeration mechanisms – allowing existing servers to be on the fabric without changing the firmware or software.

ExpressFabric achieves this new, unique capability by having the host enumeration accesses redirected to the mCPU upon initialization. The mCPU, which is the actual root complex of the network, provides enumeration responses to the host that are "appropriate", but instead of providing the real topology, they give the host a topology that the mCPU creates. When the enumeration is complete, the host has a topology in its memory that has been synthesized for it by the mCPU. The host then knows about the other ports in a manner that allows normal operation.

Host-to-Host Communication

ExpressFabric offers a range of flexible options for hosts to communicate with high performance and low latency. And to do so using standard mechanisms and application software.

Building High Performance SSD-Based Systems

One of the major enabling capabilities of ExpressFabric is the universal nature of PCIe. And nowhere is this more true than in the case of storage systems. PCIe has been used for many years throughout the storage market as a major interconnection mechanism, and there is history and momentum with the architecture and a large software application base. In storage, PCIe is the incumbent.

The newest, and the fastest growing, segment of the storage market is Solid State Discs (SSDs), and the industry has standardized on PCIe as the primary connection to this new, high performance storage element. ExpressFabric allows PCIe-based SSDs to be directly connected to the fabric with no latency-inducing translation devices. And it furthermore allows them to be connected to other PCIe-based devices and subsystems – basically everything else in a system.

In addition to this basic ability to interoperate with the rest of the system, ExpressFabric offers the extended reliability that is now part of the PCIe specification.

Most servers have difficulty handling serious errors, especially when an end-point disappears from the system due to, for example, a cabled being pulled. The problem tends to proliferate through the system until recovery becomes impractical. Downstream Port Containment (DPC) allows a downstream link to be disabled after an uncorrectable error. This makes error recovery feasible with the appropriate software, and is especially critical in storage systems, since the removal of a drive needs to be handled in a controlled and robust manner.

In addition to offering this PCI-SIG ECN, ExpressFabric devices track outstanding reads to downstream ports, and synthesize a completion so that the host does not get a completion time-out if the end-point is removed.

Highly Scalable, Flexible Fabric Topologies

ExpressFabric interfaces with standard PCIe devices, using standard software. Once inside the fabric, however, the information is routed through a Global ID (GID), rather than an address. The mapping between the address and the GID is provided by the management CPU (mCPU), and this approach allows the fabric to eliminate the hierarchical topology restriction of standard PCIe.

ExpressFabric allows other topologies such as mesh, fat tree, and many others. And it does this while allowing the components to remain architecturally and software compatible with standard PCIe.

ExpressFabric allows topologies that have more than one path between the elements of the system, and this enables the devices in the system to invoke spread routing, where the data can be sent through more than one path. Congestion information is also shared between devices, allowing the source of the data to make better decisions on how to handle the routing.

Mainstream Applications for ExpressFabric

ExpressFabric-based products are an outstanding solution when designing a heterogeneous system where there is a requirement for a flexible mix of processors, storage elements, and communication devices.

Rapid and Effective Development Tool Suite

One of the most important aspects of a technology is how quickly it can be brought to market. And PLX makes ExpressFabric quick and easy to design and deploy with the FabricBuilder suite. FabricBuilder includes:

The software that PLX includes with the package is provided in source form. This enables the designer to modify and enhance the functions of the solution.

ExpressFabric: A Fabric for the Next Generation

ExpressFabric offers the advantages of PCIe, and includes the enhancements necessary to create a state-of-the-art fabric.

Since ExpressFabric is based on PCI Express, it offers a platform that offers high volume, mainstream price points.

Relevant Videos

More Videos...

Published Articles