Globus Compute with Integrated Research Infrastructure (IRI) workflows

As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and I will give a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.

1. 1
Globus Compute with
IRI Workflows
Nick Tyler
Data & AI Services

2. NERSC Superfacility

3. 3
•National Energy Scientific Research
Scientific Computing Center
oLawrence Berkeley National Laboratory
o50th Anniversary!
oHome to Perlmutter Supercomputer
•Superfacility
oSuperfacility API
•REST API for interacting with HPC
•Now with python client!
oSpin
•Kubernetes based service platform
oDTNs and connectivity with ESNet
•Globus Data Transfers
•Sharing results with the world
NERSC and Superfacility
Download Perlmutter using Globus!

4. 4
•Multi-site feedback loops
oExpertise at experimental facility
oComputational power of HPC
•NERSC, ALCF, OLCF
oESNet to connect facilities
•Give the scientists real time feedback
oPowered by data
oAdd new techniques
•AI modeling for detector simulation
•Goal is to empower scientists!

NERSC Superfacility and IRI
Simulation
Readout
Data
Data
Analysis
Feedback
To empower researchers to meld DOE’s world-class research tools, infrastructure, and user
facilities seamlessly and securely in novel ways to radically accelerate discovery and innovation
- Ben Brown (ASCR, DOE)

5. 5
DIII-D and NERSC
•Get CAKE workflow running at NERSC
oSome parts need to be run on GA side
•Need to interact with databases
•Get readings from instruments
oUse computational power of NERSC
•ONETWO
•Ion pressure and density
•EFIT
•Equilibrium fitting code
•Current setup uses a lot of tunnels
oOpening ports can be difficult
•Depends on the site
oCall out to Globus services
•Outbound http connections

6. Globus Compute for DIII-D

7. 7
DIII-D
Control
Room
DIII-D and NERSC
Workflow
Manager
on
Compute
Slurm/PBS
Job Array
Database
DIII-D
Control
Room
Workflow
Manager
on local
Compute
Database
•Simplify cross site
communications
•Takes array of jobs and
executes each task as a
Globus Compute Task
•Still just a working proof of
concept
•Would need to be fully
integrated into workflow tool

8. 8
How it works
•Each task is a single line
oWrap each line in a
simple bash script
oRuns the script
oCapture output
•Each line is built to match
the site
omodules
opaths

9. 9
How it works
•Create object from Globus
Compute Endpoint ID
oAdd the lines to run
oRun the lines
oGet back stdout, stderr
•Run creates list of Globus
Compute Futures
oPull the results from Globus
oAs results are completed

10. 10
Future Work with IRI DIII-D and Globus
•Integrate Globus Compute into CAKE
workflow
oWaiting on funding and time
•A different workflow
oIonorb
oUsed to simulate the shot
•Predictive analysis before next shot
•Make sure the next planned shot won’t
destroy the machine
•Christine at ALCF developed Globus Flow
oDemo of Globus Flows at SC
oOptimizing it for DIII-D Workflow


LBL: Laurie Stephey, Eli Dart, Sam Williams, Pengfei Ding
ALCF: Christine Simpson, Tom Uram, Bill Allcock, Mike Papka
DIII-D: Sterling Smith, David Schissel, Torin Bechtel, Severin
Denk, Mark Kostuk, Juan Diego Colmenares