Modest scale HPC on Azure using CGYRO

Modest scale HPC on Azure using CGYRO

• 1.
  Modest scale HPCon Azure using CGYRO Igor Sfiligoi –UC San Diego Jeff Candy – General Atomics A poster at Instance Nodes Total time Comm. time Total cost NDv2 24 161s 139s $5.24 NDv2 8 369s 316s $4.00 HBv2 36 272s 104s $1.81 HBv2 18 441s 190s $1.47 HC 35 416s 113s $2.42 HC 18 763s 151s $2.28 System Nodes Total time Comm. time Summit 32 86s 67s Cori 128 165s 62s Cori 48 339s 160s Azure
• 2.
  What is CGYRO? •CGYRO is a premier tool for multi-scale plasma turbulence simulation and has been in use by the fusion community for several years • An Eulerian gyrokinetic solver and relies heavily on global FFT computations • Fusion research still very active • Several aspects of fusion energy physics are still not well understood • Experimental methods are essential for gathering new operational modes • But simulations are used to validate basic theory, plan experiments, interpret results on present devices, and ultimately to design future devices.
• 3.
  Motivation for thisexploratory work • Leadership-class HPC centers are heavily sought for and typically over-subscribed • Can we do Fusion research in other venues? • Commercial Clouds offer an appealing option • They promise immediate access to resources • All you need is $$$ • But can they deliver? • Are there true HPC-class resources available? • Can we afford it?
• 4.
  Microsoft Azure andHPC • Among the commercial Cloud providers, Microsoft Azure has the most HPC-like resources • Several instance types with Infiniband connectivity • The two most promising: • NDv2 – 8x NVIDIA V100 GPUs with 100 Gbps EDR IB • HBv2 – 120-core AMD EPYC CPUs with 200 Gbps HDR IB
• 5.
  Verifying IB performance •CGYRO is extremely network latency and throughput sensitive • One could say that it is network-bound • Microsoft Azure IB shows great characteristics using the OSU test benchmark tools Measured network latencies in us, as reported by the osu_latency tool.
• 6.
  Submit environment • UnlikeHPC centers, Microsoft Azure does not provide an HPC batch system or a shared file system out of the box • But CycleCloud as a free add-on option • CycleCloud provides several batch system options • We chose SLURM, mostly due to our familiarity with that system • Comes with ssh access and auto-scaling capabilities out-of-the-box • More advanced options require the use of their API • Initial setup relatively easy, but not trivial • Mostly a documentation issue • Hit also a couple of bugs in the advanced options (e.g. spot HPC use) (since fixed)
• 7.
  Execution environment • CycleClouddoes most of the system/batch config • Also comes with basic compiler and MPI config • However, optimized for CPU instances • No out-of-the-box GPU support • To use the GPU instances we had to do some manual changes • Create a GPU-enabled HPC VM Image, and point CycleCloud to it • Install PGI compilers (note: Now called NVIDIA HPC SDK) • Install and configure the MPI library • Comes with PGI compilers • We used the head-node NFS shared filesystem setup • Not a true HPC storage solution, but good enough for CGYRO
• 8.
  Benchmarking CGYRO onAzure – With real science • The main “benchmarking tool” was a brand new, cutting-edge CGYRO simulation: • A multi-scale simulation • N_RADIAL=1024, N_TOROIDAL=128 • https://github.com/scidac/atom-open-doc/blob/master/2020.11-SC20/multiscale_input/input.cgyro • Most of the compute time in Azure was spent advancing the progress of the above simulation • And most of that time was using Spot pricing (very little preemption incurred) • We also ran some smaller test simulations for completeness • nl03 and sh02, which represent more often used simulation profiles • These benchmark tests used a minimal fraction of total resources
• 9.
  Multi-scale benchmark resultson Azure • We started with the GPU-providing NDv2 instance (8x NVIDIA V100) • But observed that a very high fraction of the time spent in communication • We thus switched the simulation to the CPU-providing instances • HC uses “traditional” INTEL Xeon CPUs • HBv2 uses the latest AMD EPYC CPUs • Azure also has well defined per-hour price for each instance type, making for an easy cost effectiveness comparison • We focused on spot pricing which seems feasible at these scales Instance Nodes Total time Comm. time Total cost NDv2 24 161s 139s $5.24 NDv2 8 369s 316s $4.00 HBv2 36 272s 104s $1.81 HBv2 18 441s 190s $1.47 HC 35 416s 113s $2.42 HC 18 763s 151s $2.28 Slower per node, use more nodes All numbers represent one typical step during the simulation. Cost is computed using spot instance pricing.
• 10.
  Multi-scale benchmark resultson Azure • We started with the GPU-providing NDv2 instance (8x NVIDIA V100) • But observed that a very high fraction of the time spent in communication • We thus switched the simulation to the CPU-providing instances • HC uses “traditional” INTEL Xeon CPUs • HBv2 uses the latest AMD EPYC CPUs • Azure also has well defined per-hour price for each instance type, making for an easy cost effectiveness comparison • We focused on spot pricing which seems feasible at these scales Instance Nodes Total time Comm. time Total cost NDv2 24 161s 139s $5.24 NDv2 8 369s 316s $4.00 HBv2 36 272s 104s $1.81 HBv2 18 441s 190s $1.47 HC 35 416s 113s $2.42 HC 18 763s 151s $2.28 Slower per node, use more nodes AMD CPU-based HBv2 a clear winner Comparable speed to NDv2, at much lower cost All numbers represent one typical step during the simulation. Cost is computed using spot instance pricing.
• 11.
  Comparing to on-premHPC centers(Multi-scale) • To have a frame of reference, we also ran on two on-prem HPC centers we had access to • ORNL Summit – 6x NVIDIA V100 GPUs and 2x 100 Gbps IB per node • NERSC Cori – INTEL Xeon Phi (KNL) CPU and 56 Gbps IB per node • The Azure CPU instances are comparable to Cori results • Summit is significantly faster • Better networking shows Instance Nodes Total time Comm. time Total cost NDv2 24 161s 139s $5.24 NDv2 8 369s 316s $4.00 HBv2 36 272s 104s $1.81 HBv2 18 441s 190s $1.47 HC 35 416s 113s $2.42 HC 18 763s 151s $2.28 System Nodes Total time Comm. time Summit 32 86s 67s Cori 128 165s 62s Cori 48 339s 160s All numbers represent one typical step during the simulation. Cost is computed using spot instance pricing.
• 12.
  Smaller benchmark simulation– nl03 • Very similar insights when looking at the smaller nl03 test case • AMD CPU based HBv2 still the most cost effective • And an excellent alternative to Cori • NDv2 instances still network limited • Summit again scales better Instance Nodes Total time Comm. time Total cost NDv2 16 121s 92s $2.64 NDv2 4 397s 293s $2.15 HBv2 36 87s 45s $0.58 HBv2 9 289s 64s $0.48 HC 24 223s 60s $0.89 HC 12 431s 96s $0.86 System Nodes Total time Comm. time Summit 16 82s 46s Cori 64 112s 46s Cori 16 372s 120s All numbers represent one typical step during the simulation. Cost is computed using spot instance pricing.
• 13.
  Suitability of spotpricing for HPC • Spot instances definitely have downsides: • lower availability and • potential preemption during runtime • But they typically cost 66% - 88% less than ”normal”, i.e. on-demand • CGYRO can deal with occasional preemption • Using checkpointing every couple hours, with minimal overhead • At smaller node counts, we typically experienced at most a couple preemptions per day • But it does get worse with node count • And were not able to reliably exceed 24x NDv2 or 36x HBv2
• 14.
  Summary and conclusions •We explored the feasibility of running CGYRO on Azure HPC resources • With an emphasis on using them in spot mode • We observed both that • CPU-only resources were very efficient, and • that running in spot mode was doable, with minimal side effects. • The GPU-enabled resources were less cost effective but allowed for higher scaling • When Cloud budget is available, Azure is an excellent place for CGYRO
• 15.
  Acknowledgements • This presentationis based on the poster accepted at and presented at SC20 https://sc20.supercomputing.org/presentation/?id=rpost106&sess=sess337 • The creation of this presentation was supported by the U.S. Department of Energy under awards DE-FG02-95ER54309 (General Atomics Theory grant) and DE-SC0017992 (AToM SciDAC-4 project). Computing resources were provided by the Oak Ridge Leadership Computing Facility under Contract DE-AC05-00OR22725 (ALCC program) and the National Energy Research Scientific Computing Center under Contract DE-AC02-05CH11231
• 16.
  Updates since theposter was created • Microsoft Azure announced a few new improvements • A new GPU-based HPC instance • NDV4 – 8x NVIDIA A100 with 8x 200 Gbps IB (1.6 Tbps total) per node • https://azure.microsoft.com/en-us/blog/bringing-ai-supercomputing-to-customers/ • An updated version of CycleCloud • https://techcommunity.microsoft.com/t5/azure-compute/azure-cyclecloud-8-1-is-now- available/ba-p/1898011