Welcome to “Using Python in an HPC environment” course material

Content

• This course aims to give a brief, but comprehensive introduction to using Python in an HPC environment.

• You will learn how to

  • use modules to load Python

  • find site installed Python packages

  • install packages yourself

  • use virtual environments,

  • write a batch script for running Python

  • use Python in parallel

  • use Python for ML and on GPUs.

• This course will consist of lectures interspersed with hands-on sessions where you get to try out what you have just learned.

• We aim to give this course in spring and fall every year.

Schedule Spring 2025

Day	Language
Thursday 24 April	Intro to Python
Friday 25 April	Packages and basic analysis
Monday 28 April	Advanced analysis and batch jobs
Tuesday 29 April	PArallelism, GPUs and Machine Learning

Warning

Not covered

• Improve Python coding skills

• Specifics of other clusters

Course approach to deal with multiple HPC centers

Target group

• The course is for present or presumptive users at The NAISS resources at PDC and NSC as well as UPPMAX, HPC2N, LUNARC, or possibly other clusters in Sweden.

• The course is a NAISS cooperation with teachers from UPPMAX (Rackham, Snowy, Bianca), HPC2N (Kebnekaise), and LUNARC (Cosmos) and will focus on systems at the Swedish academic HPC centres with NAISS personnel.

• Although there are differences we will only have few seperate sessions.

• Most participants will use The NAISS resources Tetralith system at NSC or Dardel system at PDC for the course.

  • Alvis at C3SE (Chalmers) may be added in the future.

• Users with the “local” affiliation below, can work at the following clusters:

  • Kebnekaise: UmU, IRF, MIUN, SLU, LTU.

  • Cosmos: LU.

  • Rackham/Snowy: UU.

• The general information given in the course will be true for all/most HPC centres in Sweden.

  • The examples will often have specific information, like module names and versions, which may vary. What you learn here should help you to make any changes needed for the other centres.

  • When present, links to the Python documentation at other NAISS centres are given in the corresponding session.

Cluster documentation of Python

• Documentation at the HPC centres

  • UPPMAX

  • HPC2N

  • LUNARC

  • NSC

  • PDC

Some practicals

• Code of Conduct

  • Be nice to each other!

• Zoom

  • You should have gotten an email with the links

  • Zoom policy:

    • Zoom chat (maintained by co-teachers):

      • technical issues of zoom

      • technical issues of your settings

      • direct communication

      • each teacher may have somewhat different approach

    • collaboration document (see below):

      • “explain again”

      • elaborating the course content

      • solutions for your own work

  • Recording policy:

    • All lectures and demos will be available on HPC2Ns Youtube channel after the course.

    • Due to different teaching styles, the teacher of a session decides if a lesson will be:

      • recorded without learners

      • recorded live with learners

    • The teachers will be clear if it is recorded or not and it will be visible in the ZOOM.

    • For the live-recordings

      • The questions asked per microphone during these sessions will be recorded

      • If you don’t want your voice to appear:

        • use the collaboration document (see below)

  • The Zoom main room is used for most lectures

  • Some sessions use breakout rooms for exercises, some of which use a silent room

Q/A collabration document

• Use the Q/A page for the workshop with your questions.

  • FIX

• Use this page for the workshop with your questions

• It helps us identify content that is missing in the course material

• We answer those questions as soon as possible

Warning

• Please be sure that you have gone through the **pre-requirements**

  • It mentions the familiarity with the LINUX command line.

  • The applications to connect to the clusters: terminals and ThinLinc (remote graphical desktop)

• This course does not aim to improve your coding skills. Rather you will learn to understand the ecosystems and navigations for the the different languages on a HPC cluster.

Content of the course

Pre-requirements:

• Pre-requirements
  • Coding
• Prepare the environment
  • Log in to one of the HPC systems covered in this course
  • Text editors on the Clusters

Common:

• Schedule
  • Course dates
  • Day 1: Log in and use Basic Python in a cluster environment
  • Day 2: Use and install packages with pip and conda in isolated environments and doing basic analysis interactively
  • Day 3: Deeper analysis with pandas and seaborn, HPC usage with big data.
  • Day 4: HPC usage with parallelisms and Machine Learning with and without GPUs.
• Log and other preparations
  • Step 1: Log in!
  • Test an editor
  • Download and extract the tarball with exercises
• Use the tarball with exercises
  • Procedure
• Use a text editor
• HPC clusters
  • The HPC centers UPPMAX, HPC2N, LUNARC, NSC and Dardel
  • Briefly about the cluster hardware and system at UPPMAX, HPC2N, LUNARC, NSC and PDC
• NAISS projects overview

Lessons day 1 (Intro to Python):

• Link to Day 1 (Intro to Python)

Lessons day 2 (packages and analysis):

• Introduction
• Load and run Python and use packages
  • Check for Python versions
  • Load a Python module
  • Run
  • Packages/Python modules
  • Check current available packages
  • Demo/Type-along
  • Exercises (C. 10 min)
• Install packages
  • Introduction
• Use isolated environments
  • Isolated environments
  • Example
  • Install packages to the virtual environment with pip
  • Creator/developer
  • User
  • Prepare fore the course environments
• Interactive work on the compute nodes
  • Introduction
  • The different way HPC2N, UPPMAX, LUNARC, and NSC provide for an interactive session
  • Start an interactive session
  • End an interactive session
  • Check to be in an interactive session
  • Check that the number of cores booked is correct
  • Running a Python script in an interactive session
  • Exercises
  • Conclusion
• Desktop On Demand
  • What is Desktop On Demand?
  • When should I use it?
  • Getting Started
• Loading IDEs
  • Jupyter
  • VS Code
  • Spyder
• Matplotlib in 60 Minutes
  • Load and Run
  • Basic Terms and Application Programming Interface (API)
  • Standard Available Plot Types
  • Formatting and Placing Plot Elements
  • Key Points
• Summary
• Python documentations at the different HPC centres

Lessons day 3 (advanced analysis):

• Intro to Pandas
  • Load and Run
  • Basic Data Types and Object Classes
  • Input/Output and Making DataFrames from Scratch
  • Inspection, Cleaning, Sorting, and Merging
  • Intro to GroupBy Objects
  • Operations
  • Advanced Topics
  • Key Points
• A Brief Introduction to the Seaborn Statistical Plotting Library
  • Load and Run Seaborn
  • Common Features
  • Plotting with Seaborn
  • Key Points
• Running Python in batch mode
  • Briefly about the cluster hardware and system at UPPMAX, HPC2N, LUNARC, and NSC
  • Running your programs and scripts on UPPMAX, HPC2N, LUNARC, and NSC
  • Useful commands to the batch system
  • Example Python batch scripts
  • Exercises
• Big data with Python
  • To cover
  • Files formats
  • Allocating RAM
  • Dask
  • Exercises
• Summary day2

Lessons day 4 (parallel and ML):

• Parallel computing with Python
  • Serial optimizations
  • Threads
  • Implicit Threaded
  • Distributed
  • MPI
  • Monitoring resources’ usage
  • Exercises
• Using GPUs with Python
  • Introduction
  • GPUs on UPPMAX, HPC2N, LUNARC, and NSC systems
  • Numba example
  • Exercises
  • Additional information
• Machine Learning and Deep Learning
  • Introduction
  • Comparison of ML/DL Libraries
  • Scikit-Learn
  • PyTorch and TensorFlow
  • Miscellaneous examples
  • Exercises

Extra:

• Other courses
  • Material for improving your Python skills
  • Material for improving your general programming skills
• More about packages
  • Using setup.py
• Developing in isolated environments
• More about interactive work
  • Jupyter on UPPMAX on compute nodes
• Jupyter at Kebnekaise
  • Running JupyterLab at HPC2N - extra Python package
  • Links
• On UPPMAX clusters
  • UPPMAX clusters
  • UPPMAX packages
• On Kebnekaise cluster
  • Accessing Kebnekaise
  • Loading Python
  • Starting Python and running Python scripts
• On Bianca cluster