Skip to content

Utilizing Linux NUMA in CMM-D: Memory Tiering

  • mail
Utilizing Linux NUMA in CMM-D: Memory Tiering
Utilizing Linux NUMA in CMM-D: Memory Tiering
For the first time in the industry, Samsung Electronics has verified1 CXL* Memory Module-DRAM (CMM-D) memory operation in a real user environment last December through collaboration with Red Hat, open source software provider, leading the expansion of its CXL ecosystem. This validation enables the utilization of Non-Uniform Memory Access (NUMA*) architecture in CMM-D, extending its functionality beyond conventional DRAM modules. To make it easier for users to utilize CMM-D, Samsung Electronics and Red Hat are providing Red Hat Enterprise Linux (RHEL) CXL usage guides through the Red Hat website.2
• Compute Express Link (CXL): An open standard for high-speed, high capacity central processing unit (CPU)-to-device and CPU-to-memory connections, designed for high performance data center computers. • Non-Uniform Memory Access (NUMA): A computer memory design used in multiprocessing, where the memory access time depends on the memory location relative to the processor.
This article intends to provide an in-depth exploration of Memory Tiering, a feature enabled by the extended NUMA architecture of CMM-D, enabling easy and efficient usage in servers. We'll explore its detailed features and characteristics. Memory Hierarchy Memory Hierarchy refers to the hierarchical organization of various types of memory used in a system. This structure is distinguished by factors such as performance, capacity, and latency, with the goal of optimizing system performance by combining memory modules with similar characteristics. This enables fast access to data and efficient management. Memory can be categorized into Main Memory, Cache Memory with high speed characteristics and Memory Expander with high capacity characteristics.
Memory Hierarchy
Memory Hierarchy
• Main Memory: The primary memory composed of DRAM. The capacity is limited, but it offers fast processing speed. It serves as a temporary storage space for processes and data to be executed by CPU. • Memory Expander (CMM-D): A memory module configured via the CXL interface. It offers a higher capacity compared to main memory, although it operates at a slower processing speed. Efficiency is enhanced through Memory Tiering. • Storage: Consists of non-volatile storage devices such as SSDs, HDDs, etc. It has a large capacity at the TB level, but significantly slower speed and high latency compared to memory.

Memory Tiering When CMM-D is added on a server, it is recognized as Zero CPU NUMA (NUMA without CPU), and DIMMs are recognized as different NUMA nodes, causing differences in CPU access time. Memory Tiering is the technology that optimizes memory by considering the different access speeds of each node. It can be enabled at the kernel level or configured by using Ramdisk and Hugepage. Memory Promotion, Demotion Memory Tiering features two primary functions: Memory Promotion, which involves transferring frequently accessed hot data to high-speed DRAM memory, and Memory Demotion, which relocates less frequently used cold data to high-capacity CMM-D. Through data migration, the system can efficiently utilize the arranged data, enhancing operational efficiency.

The following are some practical examples of Memory Tiering with CMM-D. ■ Memory Tiering Configuration To configure the dax device system-ram type, download cxl, dax, and NUMA related packages. Upon checking the dax device list, you can notice that CMM-D is recognized as a normal type system-ram.
code image 1
code image 1
To activate the NUMA Demotion feature, set the numa_balancing Memory Tiering mode. Activating this option avoids excessive load on the local memory by migrating data to CMM-D.
code image 2
code image 2
※ Supported options for numa_balancing: 0: NUMA_BALANCING_DISABLED 1: NUMA_BALANCING_NORMAL 2: NUMA_BALANCING_MEMORY_TIERING
■ Memory Tiering Configuration Verification To verify if NUMA Memory Tiering is properly configured, you can use the Stress tool. The Stress tool is provided by the EPEL Repository. Download the EPEL package and then install the Stress tool package. After installing the Stress benchmark tool, perform a load exceeding 70% of memory capacity to use Swap memory.
code image 3
code image 3
Without setting Memory Tiering after adding CMM-D, if Memory Tiering is not configured and the system experiences memory overload, Swap memory is utilized. However, in an environment where Memory Tiering is configured, operations such as pgpromote_success, pgdemote_kswapd, pgmigrate_success are observed to occur through CMM-D, rather than utilizing Swap memory.
code image 4
code image 4
※ pgpromote_success: This value indicates the number of successful page promotions. Page promotion is the process of moving pages of memory to a higher level of cache or memory for faster data access. ※ pgdemote_kswapd: This value indicates the number of times kswapd. Page demotion represents the process of moving pages of memory to a lower level of cache or memory. Kswapd demotes pages or swaps them out in situations of memory shortage to free up more available memory.
1 https://semiconductor.samsung.com/news-events/news/exceptional-scalability-with-cxl-memory-samsung-and-red-hat-expand-the-ecosystem/ 2 https://www.redhat.com/ko/resources/rhel-cxl-usage-guide
Related contents
Related Contents