Hugepages

Description

 

Linux OS manages memory using pages, where physical memory is partitioned and accessed using the basic page unit. The default page size is 4096 Bytes in the x86 architecture. The mapping of these virtual memory pages to actual physical memory addresses is bufferred in "Transaction Lookaside Buffers" (TLB) with fixed amount of entries. When utilizing a huge amount of physical memory with the default pages size, the program can consume the TLB and adds processing overhead. Hugepages allows page size to be bigger than the deault 4096 Bytes, therefore hugepages can improve memory performance by increasing the maximum size of data and text that can be used in a program and reducing the TLB misses  when accessing large data sets. On Cray machines, uGNI and MPI implementation use huge pages for its internal buffers, so increasing the default page size can also increase the size of message buffers accessible by the high speed network.

 

When to Use Huge Pages

  • For SHMEM applications, map the static data and/or private heap onto huge pages.

  • For applications written in Unified Parallel C, Co-array Fortran, and other languages based on the PGAS programming model, map the static data and/or private heap onto huge pages.

  • For MPI applications, map the static data and/or heap onto huge pages.

  • For an application which uses shared memory, which needs to be concurrently registered with the high speed network drivers for remote communication.

  • For an application doing heavy I/O.

  • To improve memory performance for common access patterns on large data sets.

  • To run ISV applications with large static data that is referenced over the HSN. This applies to CCM2 (Cluster Compatibility Mode).

When to Avoid Using Huge Pages

Applications sometimes consist of many steering programs in addition to the core application. Applying huge page behavior to all processes would not provide any benefit and would consume huge pages that would otherwise benefit the core application. See HUGETLB_RESTRICT_EXE described in ENVIRONMENT VARIABLES of manpage "intro_hugepages".

 

 

How to use hugepages

 

To use hugepages, load one of the 9 available hugepages modules below before running compilers. Loading the module sets the default huge pagesize to 2M, 8M, 16M, or 64M.   No changes are required at compile and link time.  The loaded module sets the proper library paths and libhugetlbfs will be linked to binary automatically.

The following command lists the available huge pages:

			$ module avail craype-hugepages

----------------------------- /opt/cray/craype/2.5.8/modulefiles ------------------------------
craype-hugepages128K craype-hugepages2M   craype-hugepages64M
craype-hugepages16M  craype-hugepages512K craype-hugepages8M

 

Note, although larger hugepage modules are available, hugepages larger than 64M are not supported on our Cray system.

Examples

 

To use 2MB page size:

% module load craype-hugepages2M

% cc -o mycode mycode.c

 

It is recommended that the specific module also be added to the corresponding job batch file.

 

. /opt/modules/default/init/bash # Needed to add module commands to shell

module load craype-hugepages2M

aprun -n XYZ ./mycode

 

For more information please see hugetlbfs manpage (man intro_hugepages) and the hugepages discussion in the Cray Programming Environment User Guide on  the Cray documenation website.