Cluster Resource Overview

Physical Hardware Specifications

Overall Information

• Compute Nodes: 84 - 4,576 Cores, 41TB RAM

• GPU Nodes: 9 - 34 GPU Cards, 214,752 CUDA cores, 576 Cores, 8.75TB RAM

• Persistent Storage: 1PB TB

• Login Nodes: 2 - 128 Cores, 1.5TB RAM

• Education Cluster: 16 - 704 Cores, 3TB RAM

Compute Nodes

• 34 - Penguin Computing Altus Nodes - 2,176 cores, 31.7 RAM total:

  • 4 - Zen4 Nodes with 64 cores, 2.3TB RAM, 1.8TB Scratch

  • 30 - Zen4 Nodes with 64 Cores, 768GB RAM, 1.8TB Scratch

• 50 - Cisco UCS B200M5 Nodes - 2,400 cores, 9.375TB RAM total:

  • 24 - Cascade Lake Blades with 44 cores – 192GB RAM, 425GB Scratch

  • 26 - Cascade Lake Blades with 48 cores – 192GB RAM, 425GB Scratch

GPU Nodes

For more in-depth GPU information, see the Cluster GPU Overview page, or feel free to contact the HPC Team.

• 1 - Penguin Computing Altus (64 cores):

  • 2 - ** H200 80GB** GPU Cards

  • 16,896 CUDA cores per card (33,792 per node)

  • 2TB RAM

  • 6.8TB Scratch

• 2 - Penguin Computing Altus (64 cores):

  • 4 - ** A100 80GB** GPU Cards

  • 6,190 CUDA cores per card (24,768 per node)

  • 768GB RAM

  • 1.8TB Scratch

• 2 - Penguin Computing Altus (64 cores):

  • 4 - ** A100 40GB** GPU Cards

  • 6,190 CUDA cores per card (24,768 per node)

  • 1TB RAM

  • 1.8TB Scratch

• 4 - Penguin Computing Altus (64 cores):

  • 4 - ** V100 32GB** GPU Cards

  • 5,120 CUDA cores per card (20,480 per node)

  • 1TB RAM

  • 1.8TB Scratch

Persistent Storage

• VAST DataStore Storage Arrays (600GB + 400GB shared via NFS):

  • Two identical arrays, primary can grow up to 1.2PB, secondary up to 600TB, totaling up to 1.8PB without new hardware.

  • Technical Specs

  • CBox Hardware Datasheet

  • DBox Hardware Datasheet

  • Storage Switch Datasheet

• Disaggregated Share Everything (DaSE) storage, allows scaled performance and efficient cost.

  • VAST Database Overview

  • Storage Efficiency includes Data compression, deduplication, and similarity-based data reduction.

  • Nightly and weekly snapshots for restore

  • Cloud backup enabled

• Amazon Glacier Cloud Storage (unlimited)

  • Unlimited cloud storage with variable data expiration.

  • Host of cluster archive storage

Login Nodes

• 2 - Penguin Computing Altus Nodes (Newer AMD Architecture):

  • AMD Zen4 CPUs with 64 cores

  • 768GB RAM

  • 1.8TB Scratch

Education Cluster

Dedicated legacy hardware, not covered by support, for classes and academic learning. Help ensure availability for assignments and prevent impact to ongoing research.

• 16 - Cisco UCS B200M5 Nodes - 704 cores, 3TB RAM total:

  • 16 - Sky Lake Blades with 44 cores – 192GB RAM, 425GB Scratch

Cluster Architectural Features

Login Node Features

• Users connect to and interact with the cluster via login nodes.

• The login nodes have the exact same software installations as the cluster compute nodes plus some additional interactive tools for end user environments.

• There are three login nodes are available, 2 matching newer Penguin Computing AND Compute Nodes, and 1 matching the old Cisco UCS Intel Compute Nodes.

Compute Nodes Features

• Because the DEAC Cluster is not homogenous, there are multiple compute nodes grouped into different categories by type.

  • All new nodes added to the DEAC Cluster will be provided by ‘Penguin Computing’.

    • They can be identified by their use of ‘AMD’ CPUs and 25G network connection.

    • They are named ‘cpu-amd-##’

  • All ‘Cisco UCS’ nodes are legacy hardware, some portion of which will remain on DEAC until 2027.

    • They can be designated by their use of ‘Intel’ CPUs and 10G network connection.

    • They are named ‘cpu-intel-##’.

• Penguin Computing nodes are logically seperated from Cisco UCS nodes.

  • Jobs will not run between cpu-amd-## and cpu-intel-## nodes by default.

• Parallel processing jobs are strongly encouraged to request that all nodes for a job belong to a single category or processor type.

SLURM Node Features

• All compute nodes have assigned features within SLURM. These features can be specified as constraints to limit node selection for jobs. They are:

  • login: These nodes are used to submit jobs and are not assigned to any partition to execute jobs.

  • amd : These nodes contain amd cores (64-core)

  • zen# : This designates the revision of amd core architecture (the higher the number, the newer the architecture).

  • intel : These nodes contain intel cores

  • skylake : These nodes have Intel’s Xeon E5 Skylake based processors (44-core UCS nodes)

  • cascade : These nodes have Intel’s Xeon Gold Cascade Lake based processors (44 and 48-core UCS nodes)

  • rocky9 : Designates the operating system installed on the node.