Using Python for Machine Learning jobs

Questions

  • Which machine learning tools are installed at HPC2N and UPPMAX?

  • How to start the tools at HPC2N and UPPMAX

  • How to deploy GPU:s at HPC2N and UPPMAX?

Objectives

  • Get general overview of installed Machine Learning tools at HPC2N and UPPMAX

  • Get started with Machine learning in Python

  • Code along and demos (Kebnekaise and Snowy)

While Python does not run fast, it is still well suited for machine learning. However, it is fairly easy to code in, and this is particularly useful in machine learning where the right solution is rarely known from the start. A lot of tests and experimentation is needed, and the program usually goes through many iterations. In addition, there are a lot of useful libraries written for machine learning in Python, making it a good choice for this area.

Some of the most used libraries in Python for machine learning are:

  • PyTorch

  • scikit-learn

  • TensorFlow

These are all available at UPPMAX and HPC2N.

In this course we will look at two examples: PyTorch and TensorFlow, and show how you run them at our centres.

There are some examples for beginners at https://machinelearningmastery.com/start-here/#python and at https://pytorch.org/tutorials/beginner/pytorch_with_examples.html

PyTorch

PyTorch has:

  • An n-dimensional Tensor, similar to numpy, but can run on GPUs

  • Automatic differentiation for building and training neural networks

The example we will use in this course is taken from the official PyTorch page: https://pytorch.org/ and the problem is of fitting \(y=sin⁡(x)\) with a third order polynomial. We will run an example as a batch job.

You can find the full list of examples for this problem here: https://pytorch.org/tutorials/beginner/pytorch_with_examples.html

In order to run this at HPC2N/UPPMAX you should either do a batch job or run interactively on compute nodes. Remember, you should not run long/resource heavy jobs on the login nodes, and they also do not have GPUs if you want to use that.

This is an example of a batch script for running the above example, using PyTorch 1.10.0 and Python 3.9.5, running on GPUs.

TensorFlow

The example comes from https://machinelearningmastery.com/tensorflow-tutorial-deep-learning-with-tf-keras/ but there are also good examples at https://www.tensorflow.org/tutorials

We are using Tensorflow 2.6.0-CUDA-11.3-1 (and Python 3.9.5) at HPC2N (also 3.9.5 at UPPMAX).

Since there is no scikit-learn for these versions, we have to install that too:

Installing scikit-learn compatible with TensorFlow version 2.7.1 and Python version 3.10.4

  • Load modules: module load GCC/11.2.0 OpenMPI/4.1.1 SciPy-bundle/2021.10 TensorFlow/2.7.1

  • Activate the virtual environment we created earlier: source <path-to-install-dir>/vpyenv/bin/activate

  • pip install --no-cache-dir --no-build-isolation scikit-learn

We can now use scikit-learn in our example.

In order to run the above example, we will create a batch script and submit it.

Example batch script for Kebnekaise, TensorFlow version 2.6.0 and Python version 3.9.5, and the scikit-learn we installed

#!/bin/bash
# Remember to change this to your own project ID after the course!
#SBATCH -A hpc2nXXXX-YYY
# We are asking for 5 minutes
#SBATCH --time=00:05:00
# Asking for one V100
#SBATCH --gres=gpu:v100:1

# Remove any loaded modules and load the ones we need
module purge  > /dev/null 2>&1
module load GCC/10.3.0 OpenMPI/4.1.1 SciPy-bundle/2021.05 TensorFlow/2.6.0-CUDA-11.3.1

# Activate the virtual environment we installed to
source <path-to-install-dir>/vpyenv/bin/activate

# Run your Python script
python <my_tf_program.py>

Submit with sbatch <myjobscript.sh>. After submitting you will (as usual) be given the job-id for your job. You can check on the progress of your job with squeue -u <username> or scontrol show <job-id>.

Note: if you are logged in to Rackham on UPPMAX and have submitted a GPU job to Snowy, then you need to use this to see the job queue:

squeue -M snowy -u <username>

General

You almost always want to run several iterations of your machine learning code with changed parameters and/or added layers. If you are doing this in a batch job, it is easiest to either make a batch script that submits several variations of your Python script (changed parameters, changed layers), or make a script that loops over and submits jobs with the changes.

Running several jobs from within one job

This example shows how you would run several programs or variations of programs sequentially within the same job:

Example batch script for Kebnekaise, TensorFlow version 2.6.0 and Python version 3.9.5

#!/bin/bash
# Remember to change this to your own project ID after the course!
#SBATCH -A hpc2nXXXX-YYY
# We are asking for 5 minutes
#SBATCH --time=00:05:00
# Asking for one V100
#SBATCH --gres=gpu:v100:1
# Remove any loaded modules and load the ones we need
module purge  > /dev/null 2>&1
module load GCC/10.3.0 OpenMPI/4.1.1 SciPy-bundle/2021.05 TensorFlow/2.6.0-CUDA-11.3-1
# Output to file - not needed if your job creates output in a file directly
# In this example I also copy the output somewhere else and then run another executable (or you could just run the same executable for different parameters).
python <my_tf_program.py> <param1> <param2> > myoutput1 2>&1
cp myoutput1 mydatadir
python <my_tf_program.py> <param3> <param4> > myoutput2 2>&1
cp myoutput2 mydatadir
python <my_tf_program.py> <param5> <param6> > myoutput3 2>&1
cp myoutput3 mydatadir

Keypoints

  • At all clusters you will find PyTorch, TensorFlow, Scikit-learn

  • The loading are slightly different at the clusters
    • UPPMAX: All tools are available from the modules ml python_ML_packages python/3.9.5

    • HPC2N: module load GCC/11.3.0 OpenMPI/4.1.1 SciPy-bundle/2021.05 TensorFlow/2.6.0-CUDA-11.3.1