Cluster GPU Overview

GPU resources are a powerful tool utilized to accelerate research via massive parallelization of simple calculations. They have been essential to the explosion of AI, Machine Learning, and Large Language Model growth and utilization in research.

• CUDA Toolkit Website [1]

• Tesla Based Products [2]

• Tesla Software Development Tools [3]

CUDA Specifics

There are several CUDA modulefiles to load all starting with nvidia followed by installation version and type.

CUDA Installation Directory : /deac/opt/rocky9-noarch/nvidia/$VERSION CUDA Driver Version / Runtime Version : 11.8.0 / 12.4.1 CUDA Capability Major/Minor version number: 10.0

Job Submission

Users must submit all GPU jobs to the gpu partition, and request a minimum of one GPU to successfully start.

For GPU selection, if there is no preference, users can simply add “#SBATCH –gpus=1” to their batch job for any 1 GPU available. If a specific GPU type and amount is preferred, as it usually is, a user can specify the GPU via the gres directive, by adding “#SBATCH –gres=gpu:TYPE:#”. GPU types available on the DEAC Cluster are as follows:

• V100_32GB

• A100_40GB

• A100_80GB

• H200_141GB

More information about GPU job submission via the GRES (generic resource) configuration within SLURM can be found on SchedMD’s website. [4]

References

[1]

https://developer.nvidia.com/cuda-downloads

[2]

http://www.nvidia.com/object/tesla-servers.html

[3]

http://www.nvidia.com/object/tesla_software.html

[4]

https://slurm.schedmd.com/gres.html

[5] (1,2)

https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf

[6]

https://images.nvidia.com/content/technologies/volta/pdf/volta-v100-datasheet-update-us-1165301-r5.pdf

[7]

https://resources.nvidia.com/en-us-hopper-architecture/hpc-datasheet-sc23

Device Information

Hopper H200 141GB GPU Nodes

• H200 Spec Sheet [7]

• From gpu-h200-01 H100_141GB node:

module load nvidia/cuda12/cuda/12.8.1; /deac/opt/rocky9-noarch/nvidia/12.8.1/cuda/12.8.1/extras/demo_suite/deviceQuery
Loading nvidia/cuda12/cuda/12.8.1
  Loading requirement: compilers/gcc/12.3.0
/deac/opt/rocky9-noarch/nvidia/12.8.1/cuda/12.8.1/extras/demo_suite/deviceQuery Starting...

 CUDA Device Query (Runtime API)

Detected 2 CUDA Capable device(s)

Device 0: "NVIDIA H200 NVL"
  CUDA Driver Version / Runtime Version          12.8 / 12.8
  CUDA Capability Major/Minor version number:    9.0
  Total amount of global memory:                 143167 MBytes (150121021440 bytes)
MapSMtoCores for SM 9.0 is undefined.  Default to use 128 Cores/SM
MapSMtoCores for SM 9.0 is undefined.  Default to use 128 Cores/SM
  (132) Multiprocessors, (128) CUDA Cores/MP:     16896 CUDA Cores
  GPU Max Clock rate:                            1785 MHz (1.78 GHz)
  Memory Clock rate:                             3201 Mhz
  Memory Bus Width:                              6144-bit
  L2 Cache Size:                                 62914560 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 3 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 4 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

Device 1: "NVIDIA H200 NVL"
  CUDA Driver Version / Runtime Version          12.8 / 12.8
  CUDA Capability Major/Minor version number:    9.0
  Total amount of global memory:                 143167 MBytes (150121021440 bytes)
MapSMtoCores for SM 9.0 is undefined.  Default to use 128 Cores/SM
MapSMtoCores for SM 9.0 is undefined.  Default to use 128 Cores/SM
  (132) Multiprocessors, (128) CUDA Cores/MP:     16896 CUDA Cores
  GPU Max Clock rate:                            1785 MHz (1.78 GHz)
  Memory Clock rate:                             3201 Mhz
  Memory Bus Width:                              6144-bit
  L2 Cache Size:                                 62914560 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 3 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 199 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
> Peer access from NVIDIA H200 NVL (GPU0) -> NVIDIA H200 NVL (GPU1) : Yes
> Peer access from NVIDIA H200 NVL (GPU1) -> NVIDIA H200 NVL (GPU0) : Yes

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.8, CUDA Runtime Version = 12.8, NumDevs = 2, Device0 = NVIDIA H200 NVL, Device1 = NVIDIA H200 NVL
Result = PASS

Tesla A100 80GB GPU Nodes

• A100 Spec Sheet [5]

• From gpu-a100-0[1-2] A100_80GB node:

# module load nvidia/cuda12/cuda/12.4.1; /deac/opt/rocky9-noarch/nvidia/12.4.1/cuda/12.4.1/extras/demo_suite/deviceQuery
Loading nvidia/cuda12/cuda/12.4.1
 Loading requirement: compilers/gcc/12.3.0
/deac/opt/rocky9-noarch/nvidia/12.4.1/cuda/12.4.1/extras/demo_suite/deviceQuery Starting...

CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 4 CUDA Capable device(s)

Device 0: "NVIDIA A100 80GB PCIe" (truncated)
Device 1: "NVIDIA A100 80GB PCIe" (truncated)
Device 2: "NVIDIA A100 80GB PCIe" (truncated)
Device 3: "NVIDIA A100 80GB PCIe"
 CUDA Driver Version / Runtime Version          12.4 / 12.4
 CUDA Capability Major/Minor version number:    8.0
 Total amount of global memory:                 81038 MBytes (84974239744 bytes)
 (108) Multiprocessors, ( 64) CUDA Cores/MP:     6912 CUDA Cores
 GPU Max Clock rate:                            1410 MHz (1.41 GHz)
 Memory Clock rate:                             1512 Mhz
 Memory Bus Width:                              5120-bit
 L2 Cache Size:                                 41943040 bytes
 Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
 Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
 Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
 Total amount of constant memory:               65536 bytes
 Total amount of shared memory per block:       49152 bytes
 Total number of registers available per block: 65536
 Warp size:                                     32
 Maximum number of threads per multiprocessor:  2048
 Maximum number of threads per block:           1024
 Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
 Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
 Maximum memory pitch:                          2147483647 bytes
 Texture alignment:                             512 bytes
 Concurrent copy and kernel execution:          Yes with 3 copy engine(s)
 Run time limit on kernels:                     No
 Integrated GPU sharing Host Memory:            No
 Support host page-locked memory mapping:       Yes
 Alignment requirement for Surfaces:            Yes
 Device has ECC support:                        Enabled
 Device supports Unified Addressing (UVA):      Yes
 Device supports Compute Preemption:            Yes
 Supports Cooperative Kernel Launch:            Yes
 Supports MultiDevice Co-op Kernel Launch:      Yes
 Device PCI Domain ID / Bus ID / location ID:   0 / 10 / 0
 Compute Mode:
    < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
> Peer access from NVIDIA A100 80GB PCIe (GPU0) -> NVIDIA A100 80GB PCIe (GPU1) : Yes
> Peer access from NVIDIA A100 80GB PCIe (GPU0) -> NVIDIA A100 80GB PCIe (GPU2) : Yes
> Peer access from NVIDIA A100 80GB PCIe (GPU0) -> NVIDIA A100 80GB PCIe (GPU3) : Yes
> Peer access from NVIDIA A100 80GB PCIe (GPU1) -> NVIDIA A100 80GB PCIe (GPU0) : Yes
> Peer access from NVIDIA A100 80GB PCIe (GPU1) -> NVIDIA A100 80GB PCIe (GPU2) : Yes
> Peer access from NVIDIA A100 80GB PCIe (GPU1) -> NVIDIA A100 80GB PCIe (GPU3) : Yes
> Peer access from NVIDIA A100 80GB PCIe (GPU2) -> NVIDIA A100 80GB PCIe (GPU0) : Yes
> Peer access from NVIDIA A100 80GB PCIe (GPU2) -> NVIDIA A100 80GB PCIe (GPU1) : Yes
> Peer access from NVIDIA A100 80GB PCIe (GPU2) -> NVIDIA A100 80GB PCIe (GPU3) : Yes
> Peer access from NVIDIA A100 80GB PCIe (GPU3) -> NVIDIA A100 80GB PCIe (GPU0) : Yes
> Peer access from NVIDIA A100 80GB PCIe (GPU3) -> NVIDIA A100 80GB PCIe (GPU1) : Yes
> Peer access from NVIDIA A100 80GB PCIe (GPU3) -> NVIDIA A100 80GB PCIe (GPU2) : Yes

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.4, CUDA Runtime Version = 12.4, NumDevs = 4, Device0 = NVIDIA A100 80GB PCIe, Device1 = NVIDIA A100 80GB PCIe, Device2 = NVIDIA A100 80GB PCIe, Device3 = NVIDIA A100 80GB PCIe Result = PASS

Tesla A100 40GB GPU Nodes

• A100 Spec Sheet [5]

• From gpu-a100-0[3-4] A100_40GB node:

# module load nvidia/cuda12/cuda/12.4.1; /deac/opt/rocky9-noarch/nvidia/12.4.1/cuda/12.4.1/extras/demo_suite/deviceQuery
Loading nvidia/cuda12/cuda/12.4.1
 Loading requirement: compilers/gcc/12.3.0
/deac/opt/rocky9-noarch/nvidia/12.4.1/cuda/12.4.1/extras/demo_suite/deviceQuery Starting...

CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 4 CUDA Capable device(s)

Device 0: "NVIDIA A100-PCIE-40GB" (truncated)
Device 1: "NVIDIA A100-PCIE-40GB" (truncated)
Device 2: "NVIDIA A100-PCIE-40GB" (truncated)
Device 3: "NVIDIA A100-PCIE-40GB"
 CUDA Driver Version / Runtime Version          12.4 / 12.4
 CUDA Capability Major/Minor version number:    8.0
 Total amount of global memory:                 40326 MBytes (42285268992 bytes)
 (108) Multiprocessors, ( 64) CUDA Cores/MP:     6912 CUDA Cores
 GPU Max Clock rate:                            1410 MHz (1.41 GHz)
 Memory Clock rate:                             1215 Mhz
 Memory Bus Width:                              5120-bit
 L2 Cache Size:                                 41943040 bytes
 Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
 Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
 Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
 Total amount of constant memory:               65536 bytes
 Total amount of shared memory per block:       49152 bytes
 Total number of registers available per block: 65536
 Warp size:                                     32
 Maximum number of threads per multiprocessor:  2048
 Maximum number of threads per block:           1024
 Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
 Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
 Maximum memory pitch:                          2147483647 bytes
 Texture alignment:                             512 bytes
 Concurrent copy and kernel execution:          Yes with 3 copy engine(s)
 Run time limit on kernels:                     No
 Integrated GPU sharing Host Memory:            No
 Support host page-locked memory mapping:       Yes
 Alignment requirement for Surfaces:            Yes
 Device has ECC support:                        Enabled
 Device supports Unified Addressing (UVA):      Yes
 Device supports Compute Preemption:            Yes
 Supports Cooperative Kernel Launch:            Yes
 Supports MultiDevice Co-op Kernel Launch:      Yes
 Device PCI Domain ID / Bus ID / location ID:   0 / 193 / 0
 Compute Mode:
    < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
> Peer access from NVIDIA A100-PCIE-40GB (GPU0) -> NVIDIA A100-PCIE-40GB (GPU1) : Yes
> Peer access from NVIDIA A100-PCIE-40GB (GPU0) -> NVIDIA A100-PCIE-40GB (GPU2) : Yes
> Peer access from NVIDIA A100-PCIE-40GB (GPU0) -> NVIDIA A100-PCIE-40GB (GPU3) : Yes
> Peer access from NVIDIA A100-PCIE-40GB (GPU1) -> NVIDIA A100-PCIE-40GB (GPU0) : Yes
> Peer access from NVIDIA A100-PCIE-40GB (GPU1) -> NVIDIA A100-PCIE-40GB (GPU2) : Yes
> Peer access from NVIDIA A100-PCIE-40GB (GPU1) -> NVIDIA A100-PCIE-40GB (GPU3) : Yes
> Peer access from NVIDIA A100-PCIE-40GB (GPU2) -> NVIDIA A100-PCIE-40GB (GPU0) : Yes
> Peer access from NVIDIA A100-PCIE-40GB (GPU2) -> NVIDIA A100-PCIE-40GB (GPU1) : Yes
> Peer access from NVIDIA A100-PCIE-40GB (GPU2) -> NVIDIA A100-PCIE-40GB (GPU3) : Yes
> Peer access from NVIDIA A100-PCIE-40GB (GPU3) -> NVIDIA A100-PCIE-40GB (GPU0) : Yes
> Peer access from NVIDIA A100-PCIE-40GB (GPU3) -> NVIDIA A100-PCIE-40GB (GPU1) : Yes
> Peer access from NVIDIA A100-PCIE-40GB (GPU3) -> NVIDIA A100-PCIE-40GB (GPU2) : Yes

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.4, CUDA Runtime Version = 12.4, NumDevs = 4, Device0 = NVIDIA A100-PCIE-40GB, Device1 = NVIDIA A100-PCIE-40GB, Device2 = NVIDIA A100-PCIE-40GB, Device3 = NVIDIA A100-PCIE-40GB Result = PASS

Tesla V100 32 GPU Nodes

• V100 Spec Sheet [6]

• From gpu-v100-01 V100 32GB node:

# module load nvidia/cuda12/cuda/12.4.1; /deac/opt/rocky9-noarch/nvidia/12.4.1/cuda/12.4.1/extras/demo_suite/deviceQuery
Loading nvidia/cuda12/cuda/12.4.1
 Loading requirement: compilers/gcc/12.3.0
/deac/opt/rocky9-noarch/nvidia/12.4.1/cuda/12.4.1/extras/demo_suite/deviceQuery Starting...

CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 4 CUDA Capable device(s)

Device 0: "Tesla V100-PCIE-32GB" (truncated)
Device 1: "Tesla V100-PCIE-32GB" (truncated)
Device 2: "Tesla V100-PCIE-32GB" (truncated)
Device 3: "Tesla V100-PCIE-32GB"
 CUDA Driver Version / Runtime Version          12.4 / 12.4
 CUDA Capability Major/Minor version number:    7.0
 Total amount of global memory:                 32494 MBytes (34072559616 bytes)
 (80) Multiprocessors, ( 64) CUDA Cores/MP:     5120 CUDA Cores
 GPU Max Clock rate:                            1380 MHz (1.38 GHz)
 Memory Clock rate:                             877 Mhz
 Memory Bus Width:                              4096-bit
 L2 Cache Size:                                 6291456 bytes
 Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
 Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
 Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
 Total amount of constant memory:               65536 bytes
 Total amount of shared memory per block:       49152 bytes
 Total number of registers available per block: 65536
 Warp size:                                     32
 Maximum number of threads per multiprocessor:  2048
 Maximum number of threads per block:           1024
 Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
 Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
 Maximum memory pitch:                          2147483647 bytes
 Texture alignment:                             512 bytes
 Concurrent copy and kernel execution:          Yes with 7 copy engine(s)
 Run time limit on kernels:                     No
 Integrated GPU sharing Host Memory:            No
 Support host page-locked memory mapping:       Yes
 Alignment requirement for Surfaces:            Yes
 Device has ECC support:                        Enabled
 Device supports Unified Addressing (UVA):      Yes
 Device supports Compute Preemption:            Yes
 Supports Cooperative Kernel Launch:            Yes
 Supports MultiDevice Co-op Kernel Launch:      Yes
 Device PCI Domain ID / Bus ID / location ID:   0 / 193 / 0
 Compute Mode:
    < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
> Peer access from Tesla V100-PCIE-32GB (GPU0) -> Tesla V100-PCIE-32GB (GPU1) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU0) -> Tesla V100-PCIE-32GB (GPU2) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU0) -> Tesla V100-PCIE-32GB (GPU3) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU1) -> Tesla V100-PCIE-32GB (GPU0) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU1) -> Tesla V100-PCIE-32GB (GPU2) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU1) -> Tesla V100-PCIE-32GB (GPU3) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU2) -> Tesla V100-PCIE-32GB (GPU0) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU2) -> Tesla V100-PCIE-32GB (GPU1) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU2) -> Tesla V100-PCIE-32GB (GPU3) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU3) -> Tesla V100-PCIE-32GB (GPU0) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU3) -> Tesla V100-PCIE-32GB (GPU1) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU3) -> Tesla V100-PCIE-32GB (GPU2) : Yes

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.4, CUDA Runtime Version = 12.4, NumDevs = 4, Device0 = Tesla V100-PCIE-32GB, Device1 = Tesla V100-PCIE-32GB, Device2 = Tesla V100-PCIE-32GB, Device3 = Tesla V100-PCIE-32GB Result = PASS

• From gpu-v100-0[2-3] V100 32GB (Legacy Architecture) nodes:

# module load nvidia/cuda12/cuda/12.4.1; /deac/opt/rocky9-noarch/nvidia/12.4.1/cuda/12.4.1/extras/demo_suite/deviceQuery
Loading nvidia/cuda12/cuda/12.4.1
 Loading requirement: compilers/gcc/12.3.0
/deac/opt/rocky9-noarch/nvidia/12.4.1/cuda/12.4.1/extras/demo_suite/deviceQuery Starting...

CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 6 CUDA Capable device(s)

Device 0: "Tesla V100-PCIE-32GB" (truncated)
Device 1: "Tesla V100-PCIE-32GB" (truncated)
Device 2: "Tesla V100-PCIE-32GB" (truncated)
Device 3: "Tesla V100-PCIE-32GB" (truncated)
Device 4: "Tesla V100-PCIE-32GB" (truncated)
Device 5: "Tesla V100-PCIE-32GB"
 CUDA Driver Version / Runtime Version          12.4 / 12.4
 CUDA Capability Major/Minor version number:    7.0
 Total amount of global memory:                 32494 MBytes (34072559616 bytes)
 (80) Multiprocessors, ( 64) CUDA Cores/MP:     5120 CUDA Cores
 GPU Max Clock rate:                            1380 MHz (1.38 GHz)
 Memory Clock rate:                             877 Mhz
 Memory Bus Width:                              4096-bit
 L2 Cache Size:                                 6291456 bytes
 Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
 Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
 Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
 Total amount of constant memory:               65536 bytes
 Total amount of shared memory per block:       49152 bytes
 Total number of registers available per block: 65536
 Warp size:                                     32
 Maximum number of threads per multiprocessor:  2048
 Maximum number of threads per block:           1024
 Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
 Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
 Maximum memory pitch:                          2147483647 bytes
 Texture alignment:                             512 bytes
 Concurrent copy and kernel execution:          Yes with 7 copy engine(s)
 Run time limit on kernels:                     No
 Integrated GPU sharing Host Memory:            No
 Support host page-locked memory mapping:       Yes
 Alignment requirement for Surfaces:            Yes
 Device has ECC support:                        Enabled
 Device supports Unified Addressing (UVA):      Yes
 Device supports Compute Preemption:            Yes
 Supports Cooperative Kernel Launch:            Yes
 Supports MultiDevice Co-op Kernel Launch:      Yes
 Device PCI Domain ID / Bus ID / location ID:   0 / 197 / 0
 Compute Mode:
    < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
> Peer access from Tesla V100-PCIE-32GB (GPU0) -> Tesla V100-PCIE-32GB (GPU1) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU0) -> Tesla V100-PCIE-32GB (GPU2) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU0) -> Tesla V100-PCIE-32GB (GPU3) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU0) -> Tesla V100-PCIE-32GB (GPU4) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU0) -> Tesla V100-PCIE-32GB (GPU5) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU1) -> Tesla V100-PCIE-32GB (GPU0) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU1) -> Tesla V100-PCIE-32GB (GPU2) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU1) -> Tesla V100-PCIE-32GB (GPU3) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU1) -> Tesla V100-PCIE-32GB (GPU4) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU1) -> Tesla V100-PCIE-32GB (GPU5) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU2) -> Tesla V100-PCIE-32GB (GPU0) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU2) -> Tesla V100-PCIE-32GB (GPU1) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU2) -> Tesla V100-PCIE-32GB (GPU3) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU2) -> Tesla V100-PCIE-32GB (GPU4) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU2) -> Tesla V100-PCIE-32GB (GPU5) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU3) -> Tesla V100-PCIE-32GB (GPU0) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU3) -> Tesla V100-PCIE-32GB (GPU1) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU3) -> Tesla V100-PCIE-32GB (GPU2) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU3) -> Tesla V100-PCIE-32GB (GPU4) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU3) -> Tesla V100-PCIE-32GB (GPU5) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU4) -> Tesla V100-PCIE-32GB (GPU0) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU4) -> Tesla V100-PCIE-32GB (GPU1) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU4) -> Tesla V100-PCIE-32GB (GPU2) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU4) -> Tesla V100-PCIE-32GB (GPU3) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU4) -> Tesla V100-PCIE-32GB (GPU5) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU5) -> Tesla V100-PCIE-32GB (GPU0) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU5) -> Tesla V100-PCIE-32GB (GPU1) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU5) -> Tesla V100-PCIE-32GB (GPU2) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU5) -> Tesla V100-PCIE-32GB (GPU3) : Yes
> Peer access from Tesla V100-PCIE-32GB (GPU5) -> Tesla V100-PCIE-32GB (GPU4) : Yes

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.4, CUDA Runtime Version = 12.4, NumDevs = 6, Device0 = Tesla V100-PCIE-32GB, Device1 = Tesla V100-PCIE-32GB, Device2 = Tesla V100-PCIE-32GB, Device3 = Tesla V100-PCIE-32GB, Device4 = Tesla V100-PCIE-32GB, Device5 = Tesla V100-PCIE-32GB Result = PASS