PCIe Gen5 and Gen4 Switching Technology

HighPoint NVMe Solutions Leverage Industry Leading PCIe Switch Technology to unleash the full potential of NVMe Storage

HighPoint’s high-performance, highly-scalable PCIe Switch Technology resides within the core of each and every one of our PCIe Gen5 (5.0) and Gen4 (4.0) NVMe storage solutions.

The innovative hardware architecture incorporates Broadcom’s PEX series ICs, and enables our Gen5 and Gen4 product lines to maximize x16 lanes of bus bandwidth and directly support up to 8 devices, or as many as 32 and 960TB+ of storage via backplane connectivity.

The ICs provide each solution with x48 internal lanes; x16 lanes of dedicated bandwidth to the upstream port (connection to the host platform), and x32 lanes of downstream bandwidth to the device ports. The downstream bandwidth allocation is configurable, depending on the number of hosted devices and active workload.

HighPoint’s unique High-Performance PCIe Switching Architecture resides in the heart of each NVMe AIC, Adapter and Enclosure Solution

HighPoint NVMe product lines are more than simple SSD connectivity solutions. They have been engineered to unleash the full potential of the storage medium by maximizing data transfer speeds, minimizing latency and facilitating consistent, reliable I/O throughput.

All of this is made possible with by our particular implementation of PCIe Switching technology. HighPoint NVMe solutions are designed to operate independently; they are capable of utilizing all available PCIe lanes, and distributing that bandwidth to where it is needed most. In other words, our Switch Architecture is capable of self-bifurcation; it requires zero resource management from the motherboard side of the equation. All it needs to work its magic is a free PCIe 5.0 or 4.0 x16 slot.

Synthetic Hierarchy; Direct Resource Allocation & Hot-Swap/Hot-Plug Capability

HighPoint PCIe Switching Architecture employs a technique known as a “Synthetic Hierarchy” to isolate the host system’s operating system from any events triggered by PCIe topology changes (physical changes).

Synthetic Hierarchy facilitates a more efficient interaction between the host CPU and PCIe switch. This provides two key advantages; firstly, the ability of the solution to directly manage resource allocation to the downstream PCIe channels. Secondly, it enables the solution to provide true Hot-Swap /Hot-Plug capability.

Importantly, these advantages are present for both product lines; our NVMe RAID and NVMe Switch Adapters and Gen4 External NVMe RAID Enclosures.

Scale up to 32 Devices and over 980TB of Storage!

While each solution has a fixed number of device ports and device channels, these values do not necessarily reflect the total number of NVMe devices each solution can host. Our PCIe Gen5 Switching architecture, which is utilized by Rocket series NVMe Switch Adapters, is capable of being programmed to support as many as 32 U.2, U.3 or E3.S NVMe SSDs via backplane connectivity. This feature enables customers to tailor solutions for workflows that favor raw storage capacity over transfer speed; that’s translates into 980TB or storage capacity, with Petabyte + configurations just around the corner! When operating in this mode, the adapter will allocate x1 lanes of Gen5 bandwidth to each device.

How the Architecture Works

To explain how the architecture works and distributes PCIe resources, we’ve created a diagram of the Rocket 1608A PCIe Gen5 NVMe Switch AIC. Each solution incorporates a Broadcom PEX Switch IC, to minimize latency, optimize transfer stability, and provide x48 lanes of internal bandwidth, which the can distributed to the upstream and downstream ports

48 Lanes of Dedicated PCle Gen5 Bandwidth_v1.04_24_04_25.png

Positioned near the center of the PCB is the 48-Lane PEX IC (Point “A”).

As notated above (Point “B”), the Gen5 PCIe Switching architecture ensures x16 lanes of dedicated bandwidth is allocated to the upstream port; the PCIe connection to the motherboard.

The remaining 32 lanes are equally distributed to all eight device channels; this means x4 dedicated lanes of Gen5 bandwidth are available for each SSD!

The architecture enables HighPoint Gen5 NVMe solutions to deliver up to 64GB/s of transfer bandwidth and 56GB/s of real-world transfer performance from a single PCIe Gen5 x16 slot, even if positioned furthest away from the CPU! The same is true for Gen4 product lines; up to 32GB/s of transfer bandwidth, and 28GB/s sustained transfer speed, all from a single Gen4 x16 slot!

The advantage of PCIe Switching Technology is More than Skin Deep

PCIe Switch technology can enhance and streamline storage performance in a variety of ways; it’s not simply about maximize raw transfer speeds. HighPoint’s game-changing High-Performance PCIe Switching Architecture has been engineered to maximize the efficiency, responsiveness and integrity of NVMe storage configurations.

Highly Configurable & Scalable Storage Solutions: As discussed previously, the integrated PEX Switch IC enables our NVMe Solutions, specifically Rocket 7600 and 1600 series NVMe Adapters, to scale device support up to 32 individual NVMe SSDs (2.5” U.2/U.3 or E3.S form factors). The architecture is capable of dynamically distributing bus bandwidth as needed, to ensure hosted storage devices keep pace with the target application and workflow. For example, if the adapter has been configured to support 32 devices, the PCIe switch will reshuffle the bifurcation setting from x4 lanes to x1 lanes per NVMe device. The technology also enables our M.2 solutions to support up to 8 of today’s densest M.2 SSDs, and directly host up to 64TB of storage capacity.

Direct Cable Connectivity.png_v1.01_24_04_25.png

Superior Signal Integrity: Attaining peak performance is impressive, but ultimately useless if upstream and downstream bandwidth cannot be reliably sustained. HighPoint’s PCB design and PCIe Switching Architecture were engineered to maintain consistent transfer rates by maximizing signal integrity between each downstream (endpoint) device and the NVMe solution’s upstream connection with the host platform’s PCIe root complex. This provides multiple benefits; not only does this improve and stabilize transfer rates, it significantly reduces the risk of transmission errors and minimizes latency (more on this below).

Minimizes Latency: HighPoint’s PCIe Switch architecture was engineered to minimize latency during data transmission. Latency is the enemy of sustained transfer performance; it can interrupt the smooth transmission of data from NVMe storage media to the host platform, resulting in inconsistent transfer rates and reduced response time. Lowering latency is essential for maintaining the high-speed performance characteristics of NVMe storage in environments where multiple SSDs are hosted by a single PCIe device.

Efficient Data Routing: Storage solutions equipped with PCIe switches, such as HighPoint’s Gen5 and Gen4 NVMe product lines, intelligently route data between the host CPU and PCIe devices (SSDs in this case) to ensure available bandwidth is utilized efficiently. They do not need to rely on the host CPU to send data from one hosted device to another – the PCIe switch can handle this directly.

This can significantly reduce latency and CPU overhead.

PCle Lanes Direct from CPU_v1.01_24_04_23.png

Performance Isolation: Hosted devices are capable of operating independently. HighPoint Gen5 and Gen4 NVMe solutions directly manage the flow of data between all hosted devices to ensure that heavy I/O on one SSD will not adversely affect the performance of another.

Maximizes Compatibility and Interoperability: NVMe solutions with integrated PCIe Switching technology can be used with any industry standard x86 platforms, and are especially useful for PCIe Gen5 capable systems. HighPoint’s PCIe Switching Architecture enables our NVMe solutions to fully utilize and distribute x16 lanes host bandwidth, even in cases where a PCIe slot may be handicapped to maintain compatibility with standard PCIe devices. For example, some Gen5 motherboards may only provide Gen5 bandwidth to slots physically closest to the CPU (those typically used to host GPUs). This issue leads to our next topic – the importance of PCIe Switch technology for Gen5 storage media.

The importance of PCIe Switch Technology for Gen5 Applications

Unsurprisingly, modern Gen5 server and workstation platforms offer plenty of PCIe lanes; often 5 or more Gen5 capable x16 slots, directly connected to the host Intel or AMD CPU. However, not all PCIe devices can take advantage of these plentiful resources.

Though the technology offers twice the bandwidth of Gen4 (a whopping 64GB/s), standard Gen5 devices are more sensitive to signal degradation, and may automatically downgrade to Gen4 in order to stabilize a connection or rate of transfer. Slots furthest away from the CPU may exacerbate this issue, as longer pathways raise the likelihood of interference. As a result, some platforms will only provide a set number of PCIe slots guaranteed to deliver x16 lanes of Gen5 bandwidth for standard PCIe devices.

This is the key reason why some generic PCIe Gen5 NVMe AICs (add-in-card) are unable to deliver Gen5 transfer rates if not installed into these choice PCIe slots; those closest to the host CPU, and most often reserved for the system’s GPUs. And to compound this issue, systems that utilize desktop class CPU’s may limit direct to CPU lane count to a maximum of 28. As a result, NVMe AICs that rely on CPU bifurcation may suffer further bottlenecks, and limited to a single x16 or two x8 connections. This will effectively cut performance potential by half; the equivalent of using only two SSDs!

In contrast, a PCIe Switch NVMe AIC manages PCIe resources internally, and can distribute available bandwidth effectively across all hosted devices. This enables such devices to deliver Gen5 performance regardless of which Gen5 capable slot it is paired with. This is HighPoint has made PCIe Switching architecture a key component of every one of our Gen5 storage solutions.

HighPoint PCIe Gen5 NVMe Switch architecture was designed to addresses potential performance and stability bottlenecks by internally managing available PCIe resources. Not only has the PCB design of Rocket 7600 and 1600 series AICs and Adapters been engineered to maintain robust signal integrity, the integration of dedicated PCIe switching hardware all but eliminates these risks. HighPoint’s implementation ensures all x16 lanes of Gen5 host bandwidth are fully utilized, and dynamically distributed to where it is needed most.