High Performance Computing

Param-Kamrupa

---

PARAM Kamrupa Supercomputing facility at IIT Guwahati. It covers a wide range of topics ranging from a detailed description of the hardware infrastructure to the information required to utilize the supercomputer, such as information about logging on to the supercomputer, submitting jobs, retrieving the results on to user’s Laptop/ Desktop etc. In short, the manual describes all that one needs to know to effectively utilize PARAM Kamrupa.
The supercomputer PARAM Kamrupa is based on heterogeneous and hybrid configuration of Intel Xeon Cascade Lake processors, and NVIDIA Tesla V100 GPU cards. The system was designed and implemented by HPC Technologies team, Centre for Development of Advanced Computing (C-DAC).
It consists of 2 Master nodes, 4 Login nodes, 4 Service nodes and 156 (CPU+GPU) compute nodes with total peak computing capacity of 838 (CPU+GPU) TFLOPS performance.

Overview

---

System Hardware Specifications PARAM Kamrupa system is based on processor Intel Xeon Platinum 8268 and Intel Xeon Gold 6248 and with total peak performance of 838 TFLOPS. The cluster consists of compute nodes connected with BullSequana XH2000 HDR 100 InfiniBand interconnect network. The system uses the Lustre parallel file system.

     Total number of nodes: 166 (10 + 156)

• Master nodes: 2
• Login nodes: 4
• Service nodes: 4
• CPU only nodes: 75
• GPU ready nodes: 32
• GPU nodes: 10
• High Memory nodes:39

Master Nodes

---

PARAM Kamrupa is an aggregation of a large number of computers connected through networks. The basic purpose of the master node is to manage and monitor each of the constituent components of PARAM Kamrupa from a system’s perspective. This involves operations like monitoring the health of the components, the load on the components, the utilization of various sub-components of the computers in PARAM Kamrupa.

Specification:

• Master Nodes: 2
• CPU: 2* Intel Xeon G-6248
• Cores =40, 2.5 GHz
• Total Cores = 80 cores
• Memory= 384 GB
• Total Memory = 768 GB
• HDD = 1 TBx8

Login Nodes

---

Login nodes are typically used for administrative tasks such as editing, writing scripts, transferring files, managing your jobs, and so on. You will always get connected to one of the login nodes. From the login nodes, you can get connected to a compute node and execute an interactive job or submit batch jobs through the batch system (SLURM) to run your jobs on compute nodes. For ALL users PARAM Kamrupa login nodes are the entry points and hence are shared. By default, there will be a limit on the CPU time that can be used on a login node by a user and there is a limit/user on the memory as well. If any of these are exceeded, the job will get terminated.

Specification:

• Login Nodes: 4
• CPU: 2* Intel Xeon G-6248
• Cores = 40, 2.5 GHz
• Total Cores = 160 cores
• Memory= 384 GB
• Total Memory = 1536 GB
• HDD = 1 TBx8

Service Nodes

---

Typically, the purpose of the service node is to provide Security, Management, Monitoring and other services to the cluster.

Specification:

• Service Nodes: 4
• CPU: 2* Intel Xeon G-6248
• Cores = 40, 2.5 GHz
• Total Cores = 160 cores
• Memory= 384 GB
• Total Memory= 1536 GB
• HDD = 1 TBx5

Compute Nodes

---

CPU nodes are indeed the work horses of PARAM Kamrupa. All the CPU intensive activities are carried on these nodes. Users can access these nodes from the login node to run interactive or batch jobs. Some of the nodes have higher memory, which can be exploited by users in the aforementioned way.
GPU compute nodes have CPU cores along with accelerators cards. For some applications, GPUs get markedly high performance. For exploiting these, one has to make use of special libraries, which map computations on the Graphical Processing Units (Typically one has to make use of CUDA or OpenCL).

Specification:

CPU only Compute Nodes: 75	CPU only Compute Nodes with High memory: 39	GPU ready Compute Nodes: 32	GPU Compute Nodes: 10
2* Intel Xeon Platinum 8268 Cores = 48, 2.9 GHz Total Cores = 3600 cores Memory= 192 GB, DDR4 2933 MHz Total Memory=14400 GB SSD = 480 GB	2* Intel Xeon Platinum 8268 Cores = 48, 2.9 GHz Total Cores = 1872 Memory= 768 GB, DDR4 2933 MHz Total Memory=29952 GB SSD = 480 GB	2* Intel Xeon Platinum 8268 Cores = 48, 2.9 GHz Total Cores = 1536 cores Memory= 192 GB, DDR4 2933 MHz Total Memory=6144 GB SSD = 480 GB	2* Intel Xeon Gold-6248 Cores = 40, 2.5 GHz Total Cores = 400 Memory= 192 GB, DDR4 2933 MHz Total Memory= 1920 GB SSD = 480 GB
-----------	-----------	-----------	per node 2*Nvidia V100 per node GPU Cores per node= 2*5120= 10240 GPU Memory = 16 GB HBM2 per Nvidia V100

Storage

---

• Based on Lustre parallel file system.
• Total useable capacity of 1.0 PB Primary storage.
• Throughput 25GB/s

Operating System

---

• Operating system on PARAM Kamrupa is Linux – CentOS 7.9