Exploring the Powerhouse: A Deep Dive into PCIe Switch Chipsets

Identifying the upstream and downstream capabilities of a PCIe device is key to determining the effectiveness of the product. This is especially true for PCIe NVMe AIC (add-in-card) and Adapters. NVMe SSDs are designed to interface directly with the system CPU via the PCIe bus. Determining whether or not an NVMe solution can fully utilize the available PCIe bandwidth, and allocate this bandwidth to where it is needed most, allows one to separate the wheat from the chaff. This article explains the functionality and architecture behind the concepts of upstream and downstream, and how they relate to an NVMe-based storage solution.

Deciphering PCIe Terminology: How to identify an NVMe Solution’s Upstream & Downstream Bandwidth

First, lets’ start with the basics; a brief overview of what Upstream and Downstream bandwidth refers to, and how this is related to a PCIe add-in-card (port types and bandwidth allocation):

The illustration above represents the HighPoint’s Rocket 7628A PCIe Gen5 x16 Pro-Class RAID Adapter. It allocates x16 lanes of dedicated Upstream Bandwidth, and x4 lanes of Downstream Bandwidth to each of the four MCIO 8i ports. This distribution is ideal; the upstream and downstream bandwidth is perfectly in sync, and x4 lanes are allocated to each SSD (which ensures the product can deliver maximum throughput).

A) Upstream Port (USP): The PCIe Switch USP is used to interface with the host computing platform’s PCIe root complex (which serves as a sort of bridge between the CPU, memory and PCIe bus). The bandwidth allocated to this port is referred to the as the Upstream Bandwidth, and is generally denoted by an “x#” value, such as x8 or x16.

B) Downstream Port (DSP): DSPs interface with the PCIe endpoint devices. The bandwidth allocated to the DSPs is referred to as Downstream Bandwidth. In the context of an NVMe AIC, this refers to the NVMe SSDs.

Bandwidth Allocation: How the PCIe switch allocates bandwidth (X# of electrical lanes) to the NVMe devices (SSDs in the case of an NVMe AIC). In order to avoid a performance bottleneck when all devices are accessed, the total bandwidth allocated to the DSPs should not exceed what is allocated to the USP.

How does an AIC Distribute Bandwidth?

Ok, so we now understand that the product’s Electrical Lane bandwidth should correspond with its Upstream bandwidth. How do we determine how this bandwidth is distributed Upstream to the system, and Downstream to each NVMe SSD.

Identifying Upstream: In terms of an PCIe NVMe AIC, Upstream refers to the maximum electrical lanes the card can output to the system. Most AICs denote this by the “x#” value assigned to the product description, such as the forementioned Rocket 7628A “Gen5 x16”. HighPoint makes this easy for customers – our products deliver exactly what is stated by the product name. For a non-HighPoint solution, this is not always the case, as they may be simply referring to the card’s mechanical requirement (type of slot it will fit into). If in doubt, check the published specifications.

Identifying Downstream: As mentioned previously, in regards to an NVMe AIC, “Downstream” refers to how bandwidth is distributed to each of the AIC’s NVMe ports. Ideally, the NVMe AIC solution would be capable of allocating x4 lanes per device port. This applies to NVMe media of any generation, and enables the SSD to reach the theoretical maximum throughout.

The Upstream and Downstream capabilities of a give NVMe AIC is determined by two things; the AIC’s PCIe Switch Chipset, and AIC’s hardware architecture (how it makes use of the Switch Chipset, if present).

What is a switch chipset? A PCIe Switch chipset is chip or set of chips designed to allocate bandwidth (total number of PCIe lanes) to each “port”. Any true professional-grade, high-performance PCIe NVMe storage solution will be equipped with dedicated PCIe Switch.

When discussing PCIe Switch Chipsets, “port” can refer to an individual device port or the device itself (AIC in this case), as switch chipsets are employed by any number of computing devices (such as a motherboard, AIC/Adapter or backplane). For the purposes of this article, “port” refers to the AIC or Adapters Upstream Port (connection to the computer) and Downstream ports (NVMe device ports).

You can determine much about the capabilities of the AIC if you can identify it’s PCIe Switch chipset.

The two major players in the PCIe Switch chipset market are ASmedia and Broadcom. HighPoint NVMe solutions employ Broadcom Switch ICs.

Broadcom PEX88049

Industry’s Fastest & Most Flexible PCIe/NVMe Architecture: HighPoint PCIe Gen5 NVMe solutions employ the latest iteration of our proven High-Performance PCIe Switching Architecture, which utilizes Broadcom’s PEX89048 Switch IC. This architecture provides each AIC or Adapter with 48-lanes of internal Gen5 host bandwidth; x16 lanes of which are allocated to the upstream port, with x4 lanes dedicated to each downstream port. The is what enables our Gen5 solutions to deliver a class leading 64GB/s of transfer bandwidth, and over 60GB/s of real-world transfer performance.

Conclusion

By now, it should be clear that to truly determine the Upstream/Downstream capability of an NVMe AIC or AIC drive, you must consider the number of NVMe SSDs the target device can support, how it is able to distribute bandwidth to each of these SSDs (Downstream), and whether or not the device is able to saturate the PCIe lanes it has been allocated (Upstream).

Unlike the majority of NVMe AICs, adapters and HBAs in today’s marketplace, HighPoint NVMe solutions are engineered to take full advantage of x16 lanes of host bandwidth, and actively work to ensure none of it is wasted. Due to our unique hardware architecture, which integrates Broadcom’s leading PEX Series Switches, SSD and Rocket Series NVMe AICs, Adapters, Enclosures and AIC SSDs allocate the maximum possible host bandwidth to each NVMe device port, and deliver 60+GB/s (60,000MB/s) of real-world transfer throughput; the maximum possible via a single PCIe 5.0 slot!

Learn More about HighPoint PCIe Switching Technology