2.1.11 SMP interconnect and accelerator interface
The Power10 processor provides a highly-optimized, 32 Gbps differential signaling technology interface that is structured in 16 entities. Each entity consists of eight data lanes and one spare lane. This interface can facilitate the following functional purposes:
Ê First- or second-tier, symmetric multiprocessing link interface, enabling up to 16 Power10 processors to be combined into a large, robustly scalable, single-system image.
Ê Open Coherent Accelerator Processor Interface (OpenCAPI), to attach cache coherent and I/O-coherent computational accelerators, load/store addressable host memory devices, low latency network controllers, and intelligent storage controllers.
Ê Host-to-host integrated memory clustering interconnect, enabling multiple Power10 systems to directly use memory throughout the cluster.
Note: The OpenCAPI interface and the memory clustering interconnect are Power10 technology option for future use.
Because of the versatile nature of signaling technology, the 32 Gbps interface is also referred to as Power/A-bus/X-bus/OpenCAPI/Networking (PowerAXON) interface. The IBM proprietary X-bus links connect two processors on a board with a common reference clock. The IBM proprietary A-bus links connect two processors in different drawers on different reference clocks by using a cable.
OpenCAPI is an open interface architecture that allows any microprocessor to attach to the following items:
Ê Coherent user-level accelerators and I/O devices
Ê Advanced memories accessible through read/write or user-level DMA semantics
The OpenCAPI technology is developed, enabled, and standardized by the OpenCAPI Consortium. For more information about the consortium’s mission and the OpenCAPI protocol specification, see OpenCAPI Consortium.
The PowerAXON interface is implemented on dedicated areas that are at each corner of the Power10 processor die. The Power E1080 server makes use of this interface to implement single-drawer chip-to-chip and drawer-to-drawer chip interconnects.
The Power E1080 single-drawer chip-to-chip SMP interconnect features the following properties:
Ê Three (2 x 9)-bit on planar buses per Power10 single chip module (SCM)
Ê Eight data lanes, plus one spare lane in each direction per chip-to-chip connection
Ê 32 Gbps signaling rate providing 128 GBps per chip-to-chip SMP connection bandwidth, an increase of 33% compared to the Power E980 single-drawer implementation
64 IBM Power E1080: Technical Overview and Introduction
Ê 4-way SMP architecture implementations build out of four Power10 SCMs per drawer in 1-hop topology
The Power E1080 drawer-to-drawer SMP interconnect features the following properties: Ê Three (2 x 9)-bit buses per Power10 SCM
Ê Eight data lanes plus one spare lane in each direction per chip-to-chip connection
Ê Each of the four SCMs in a drawer is connected directly to an SCM at the same position in every other drawer in a multi-node system
Ê 32 Gbps signaling rate, which provides 128 GBps per chip-to-chip inter node SMP connection bandwidth
Ê 8-socket, 12-socket and 16-socket SMP configuration options in 2-hop topology
Figure 2-7 shows the SMP connections for a fully configured 4-node 16-socket Power E1080 system. The blue lines represent the chip-to-chip connections within one system node. The green lines represent the drawer-to-drawer SMP connections
Figure 2-7 SMP interconnect in a 4-node 16-socket Power E1080 system
From the drawing that is shown in Figure 2-7, it is easy to deduce that each socket is directly connected to any other socket within one system node and only one intermediary socket is required to get from a chip to any other chip in another CEC drawer.
Chapter 2. Architecture and technical overview 65