Big data with Python

“Learning outcomes”

Learners

  • can allocate resources sufficient to data size

  • can decide on useful file formats

  • can use data-chunking as technique

  • know where to learn more

“For teacher”

Preliminary timings. Starting at 13.00

  • Intro 10 min

  • Memory 5

  • Exercise Allocation 10

  • Files 5

  • Exercise files 10

  • Packages 5

  • Dask 5

  • BREAK 15min 13.50-14.05

  • Exercise Dask 30

High-Performance Data Analytics (HPDA)

What is it?

  • High-performace data analytics (HPDA), a subset of high-performance computing which focuses on working with large data.

    • The data can come from either computer models and simulations or from experiments and observations, and the goal is to preprocess, analyse and visualise it to generate scientific results.

  • “Big data refers to data sets that are too large or complex to be dealt with by traditional data-processing application software. […]

    • Big data analysis challenges include capturing data, data storage, data analysis, search, sharing, transfer, visualization, querying, updating, information privacy, and data source.” (from Wikipedia)

Discussion

Do you already work with large data sets?

Why we need to take special actions

Remember this one?

../_images/when-to-use-pandas.png

Discussion

  • What can limit us?

What the constraints are

  • storage

  • memory

What do we need to cover??

  • storage –> make more effective files

  • reading into memory

    • –> read just parts of files into memory

    • –> chunking

  • allocate more memory

Solutions and tools

  • Allocate enough RAM

    • If you are running ready tools

    • or cannot update code or use other packages

  • Choose file format for reading and writing

  • Choose the right Python package

  • Is chunking suitable?

Allocating RAM

  • How much is actually loaded into the working memory (RAM)

  • Is more data in variables created during the run or work?

Discussion

Have you seen the Out-of-memory (OOM) error?

What to do

  • By allocating many cores on a node will give you more available memory

  • If the order 128 GB is not enough there are so-called fat nodes with at least 512 GB and up to 3 TB.

  • On some clusters you do not have to request additional CPUs to get additional memory. Use the slurm options

    • --mem or

    • --mem-per-cpu

Important

  • You do not have to explicitely run threads or other parallelism.

  • Allocating several nodes for one one big problem is not useful.

    • Note that shared memory among the cores works within node only.

Principles

Use the Slurm options for either “BATCH”, “INTERACTIVE” from command line or from OnDemand GUIs.

  • Allocate RAM using the full node RAM divided by number of course principle

    • Ex: 128 GB with 20 cores –> 6.4 GB per core

    • Allocate number of cores to cover your needs.

    • -n <number>

  • Request the memory needed and choose number of cores

    • --mem=<size>[K|M|G|T]

    • Example: --mem=150G

  • Request the memory-per-core needed and choose number of cores

    • --mem-core=<size>[K|M|G]

    • Example: --mem-per-cpu=16G

  • Request a “FAT” node.

    • Typically you can only allocate a full node here, no core parts.

    • You ask here for a non-default partition.

    • How to do this, search your cluster documentation, see exercise below.

Note

  • “core-hours” drawn from your project may be set to the maximum of “number of cores” and “memory part of node” requested.

  • So there is no win to ask for one core but much memory!

Exercise: Memory allocation (10 min)

  1. Log in to a Desktop (ThinLinc or OnDemand) (see Log in and other preparations)

Discussion

  • Take some time to find out the answers for your specific cluster for the questions below, using the table of hardware below.

Table of hardware

Hardware

Hardware

Technology

Kebnekaise

Pelle

Bianca

Cosmos

Tetralith

Dardel

Alvis

Cores/compute node

28 (72 for largemem, 128/256 for AMD Zen3/Zen4)

48 (96 with hyperthreading/SMT)

16

48

32

128

many different (updated soon)

Memory/compute node

128-3072 GB

768-3072 GB

128-512 GB

256-512 GB

96-384 GB

256-1760 GB

many different

GPU

NVidia V100, A100, A6000, L40s, H100, A40, AMD MI100

NVidia L40s, H100, T4, A2)

NVidia A100

NVidia A100

NVidia T4

4 AMD Instinct™ MI250X á 2 GCDs

many different

Actually start an interactive sesion with 4 cores for 3 hours.

  • We will use it for the exercises later.

  • Since it may take some time to get the allocation we do it now already!

  • Follow the best procedure for your cluster, e.g. from command-line or OnDemand.

How?

The following Slurm options needs to be set

  • -t 3:0:0

  • -n 4

  • -A <proj>

  • -p <partition> may be needed in some clusters

    • Dardel: -p shared

Compute allocations in this workshop

  • Pelle: uppmax2025-2-393

  • Kebnekaise: hpc2n2025-151

  • Cosmos: lu2025-7-106

  • Alvis: naiss2025-22-934

  • Tetralith: naiss2025-22-934

  • Dardel: naiss2025-22-934

How to get a node with more RAM?

  • See local HPC center documentation in how to do so!

  • Try first to search or navigate the pages

Note

  • We recommend a desktop environment for speed of the graphics.

  • connecting from local terminal with “ssh -X” (X11 forwarding) can be be used but is slower.

File formats

Bits and Bytes

  • The smallest building block of storage and memory (RAM) in the computer is a bit, which stores either a 0 or 1.

  • Normally a number of 8 bits are combined in a group to make a byte.

  • One byte (8 bits) can represent/hold at most 2^8 distinct values. Organising bytes in different ways can represent different types of information, i.e. data.

Numerical data

Different numerical data types (e.g. integer and floating-point numbers) can be represented by bytes. The more bytes we use for each value, the larger is the range or precision we get, but more bytes require more memory.

  • DataTypes

    • 16-bit: short (integer)

    • 32-bit: Int (integer)

    • 32-bit: Single (floating point)

    • 64-bit: Double (floating point)

  • For some use cases, the precision is not that important, 1% error, or so, is not that crucial. Faster and less data storage!

Text data

  • DataTypes

    • 8-bit: char

more

  • When it comes to text data, the simplest character encoding is ASCII (American Standard Code for Information Interchange) and was the most common character encodings until 2008 when UTF-8 took over.

  • The original ASCII uses only 7 bits for representing each character and therefore encodes only 128 specified characters. Later it became common to use an 8-bit byte to store each character in memory, providing an extended ASCII.

  • As computers became more powerful and the need for including more characters from other languages like Chinese, Greek and Arabic became more pressing, UTF-8 became the most common encoding. UTF-8 uses a minimum of one byte and up to four bytes per character.

Data and storage format

In real scientific applications, data is complex and structured and usually contains both numerical and text data. Here we list a few of the data and file storage formats commonly used.

Tabular data

  • A very common type of data is “tabular data”.

  • Tabular data is structured into rows and columns.

  • Each column usually has a name and a specific data type while each row is a distinct sample which provides data according to each column (including missing values).

  • The simplest and most common way to save tabular data is via the so-called CSV (comma-separated values) file.

Gridded data

  • Gridded data is another very common data type in which numerical data is normally saved in a multi-dimensional rectangular grid. Most probably it is saved in one of the following formats:

more

  • Hierarchical Data Format (HDF5) - Container for many arrays

  • Network Common Data Form (NetCDF) - Container for many arrays which conform to the NetCDF data model

  • Zarr - New cloud-optimized format for array storage

Meta data

Metadata consists of various information about the data. Different types of data may have different metadata conventions.

more

  • In Earth and Environmental science, there are widespread robust practices around metadata. For NetCDF files, metadata can be embedded directly into the data files. The most common metadata convention is Climate and Forecast (CF) Conventions, commonly used with NetCDF data.

  • When it comes to data storage, there are many types of storage formats used in scientific computing and data analysis. There isn’t one data storage format that works in all cases, so choose a file format that best suits your data.

CSV (comma-separated values)

Best use cases: Sharing data. Small data. Data that needs to be human-readable.

  • Key features

    Type: Text format Packages needed: NumPy, Pandas Space efficiency: Bad Good for sharing/archival: Yes

more

Tidy data:

Speed: Bad Ease of use: Great

Array data:

Speed: Bad Ease of use: OK for one or two dimensional data. Bad for anything higher.

HDF5 (Hierarchical Data Format version 5)

  • HDF5 is a high performance storage format for storing large amounts of data in multiple datasets in a single file.

  • It is especially popular in fields where you need to store big multidimensional arrays such as physical sciences.

  • Best use cases: Working with big datasets in array data format.

  • Key features

    • Type: Binary format

    • Packages needed: Pandas, PyTables, h5py

    • Space efficiency: Good for numeric data.

    • Good for sharing/archival: Yes, if datasets are named well.

more

  • Tidy data:

    • Speed: Ok

    • Ease of use: Good

  • Array data:

    • Speed: Great

    • Ease of use: Good

NETCDF4 (Network Common Data Form version 4)

  • NetCDF4 is a data format that uses HDF5 as its file format, but it has standardized structure of datasets and metadata related to these datasets.

  • This makes it possible to be read from various different programs.

    Best use cases: Working with big datasets in array data format. Especially useful if the dataset contains spatial or temporal dimensions. Archiving or sharing those datasets.

  • Key features

    • Type: Binary format

    • Packages needed: Pandas, SciPy, netCDF4/h5netcdf, xarray

    • Space efficiency: Good for numeric data.

    • Good for sharing/archival: Yes.

more

  • Tidy data:

    • Speed: Ok

    • Ease of use: Good

  • Array data:

    • Speed: Good

    • Ease of use: Great

  • NetCDF4 is by far the most common format for storing large data from big simulations in physical sciences.

  • The advantage of NetCDF4 compared to HDF5 is that one can easily add additional metadata, e.g. spatial dimensions (x, y, z) or timestamps (t) that tell where the grid-points are situated. As the format is standardized, many programs can use this metadata for visualization and further analysis.

An overview of common data formats

Name:
Human
readable:
Space
efficiency:
Arbitrary
data:
Tidy
data:
Array
data:
Long term
storage/sharing:

Pickle

❌

🟨

✅

🟨

🟨

❌

CSV

✅

❌

❌

✅

🟨

✅

Feather

❌

✅

❌

✅

❌

❌

Parquet

❌

✅

🟨

✅

🟨

✅

npy

❌

🟨

❌

❌

✅

❌

HDF5

❌

✅

❌

❌

✅

✅

NetCDF4

❌

✅

❌

❌

✅

✅

JSON

✅

❌

🟨

❌

❌

✅

Excel

✅

❌

❌

🟨

❌

🟨

Graph formats

🟨

🟨

❌

❌

❌

✅

Important

Legend

  • ✅ : Good

  • 🟨 : Ok / depends on a case

  • ❌ : Bad

Adapted from Aalto university’s Python for scientific computing

See also

  • ENCCS course “HPDA-Python”: Scientific data

  • Aalto Scientific Computing course “Python for Scientific Computing”: Xarray

Exercise file formats (10 minutes)

Reading NetCDF files

View file formats

  • Go over file formats and see if some are more relevant for your work.

  • Would you look at other file formats and why?

(optional)

Computing efficiency with Python

Python is an interpreted language, and many features that make development rapid with Python are a result of that, with the price of reduced performance in many cases.

  • Dynamic typing

  • Flexible data structures

  • There are some packages that are more efficient than Numpy and Pandas.

    • SciPy is a library that builds on top of NumPy.

      • It contains a lot of interfaces to battle-tested numerical routines written in Fortran or C, as well as Python implementations of many common algorithms.

      • Reads NETCDF!

    • ENCCS course material

Xarray package

  • xarray is a Python package that builds on NumPy but adds labels to multi-dimensional arrays.

    • introduces labels in the form of dimensions, coordinates and attributes on top of raw NumPy-like multidimensional arrays, which allows for a more intuitive, more concise, and less error-prone developer experience.

    • It also borrows heavily from the Pandas package for labelled tabular data and integrates tightly with dask for parallel computing.

  • Xarray is particularly tailored to working with NetCDF files.

  • But work for aother files as well

  • Explore it a bit in the (optional) exercise below!

Dask

How to use more resources than available?

../_images/when-to-use-pandas.png

Dask is very popular for data analysis and is used by a number of high-level Python libraries:

  • Dask is composed of two parts:

    • Dask Clusters

      • Dynamic task scheduling optimized for computation. Similar to other workflow management systems, but optimized for interactive computational workloads.

      • ENCCS course

    • “Big Data” Collections

      • Like parallel arrays, dataframes, and lists that extend common interfaces like NumPy, Pandas, or Python iterators to larger-than-memory or distributed environments. These parallel collections run on top of dynamic task schedulers.

      • ENCCS course

  • Dask provides dynamic parallel task scheduling and three main high-level collections:

See also

  • dask_ml package: Dask-ML provides scalable machine learning in Python using Dask alongside popular machine learning libraries like Scikit-Learn, XGBoost, and others.

  • Dask.distributed: Dask.distributed is a lightweight library for distributed computing in Python. It extends both the concurrent.futures and dask APIs to moderate sized clusters.

  • A Dask array looks and feels a lot like a NumPy array.

  • However, a Dask array uses the so-called “lazy” execution mode, which allows one to

    • build up complex, large calculations symbolically

    • before turning them over the scheduler for execution.

Chunks

  • Dask divides arrays into many small pieces (chunks), as small as necessary to fit it into memory.

  • Operations are delayed (lazy computing) e.g.

    • tasks are queued and no computation is performed until you actually ask values to be computed (for instance print mean values).

  • Then data is loaded into memory and computation proceeds in a streaming fashion, block-by-block.

  • And data is gathered in the end.

  • Tools like Dask and xarray handle “chunking” automatically.

  • Note that number of chunks does not need to be equal to number of cores.

Big file → split into chunks → parallel workers → results combined.

To think of

  • Chunk size and number of them affect the performance due to overhad/administration of the chunking and combination.

  • Briefly explain what happens when a Dask job runs on multiple cores.

Example from dask.org

# Arrays implement the Numpy API
import dask.array as da
x = da.random.random(size=(10000, 10000),
                     chunks=(1000, 1000))
x + x.T - x.mean(axis=0)
# It runs using multiple threads on one node.
# It could also be distributed to multiple nodes

Polars package

  • polars is a Python package that presnts itself as Blazingly Fast DataFrame Library

    • Utilizes all available cores on your machine.

    • Optimizes queries to reduce unneeded work/memory allocations.

    • Handles datasets much larger than your available RAM.

    • A consistent and predictable API.

    • Adheres to a strict schema (data-types should be known before running the query).

Key features

  • Fast: Written from scratch in Rust

  • I/O: First class support for all common data storage layers

  • Intuitive API: Write your queries the way they were intended. Internally, there is a query optimizer.

  • Out of Core: streaming without requiring all your data to be in memory at the same time. I.e. chunking

  • Parallel: dividing the workload among the available CPU cores without any additional configuration.

  • GPU Support: Optionally run queries on NVIDIA GPUs

  • Apache Arrow support

https://pola.rs/

Exercises: Packages

Set up the environment

Important

Interactive use (Recommended)

Follow the instructions here: https://docs.hpc2n.umu.se/tutorials/connections/#example__-__jupyter__with__extra__modules.

  • Add these lines in the batch script

module load SciPy-bundle/2023.07 matplotlib/3.7.2 Tkinter/3.11.3

  • Continue and start Jupyter

  • And install polars, dask & xarray to ~/.local/ if you don’t already have it

! pip install --user xarray
! pip install --user dask
! pip install --user polars

  • You may have to restart the Jupyter kernel (or even Jupyter session) to be able to be able to load the just instaleld package(s).

Chunk sizes in Dask

  • The following example calculate the mean value of a random generated array.

  • Run the 2 examples and see the performance improvement by using dask.

import numpy as np

%%time
x = np.random.random((20000, 20000))
y = x.mean(axis=0)

But what happens if we use different chunk sizes? Try out with different chunk sizes:

  • What happens if the dask chunks=(20000,20000)

  • What happens if the dask chunks=(250,250)

(Optional) Polars

  • Browse: https://docs.pola.rs/.

    • find something interesting for you! Test some lines if you want to!

    • tips:

  • Check if your cluster has Polars!

Summary

Follow-up discussion

  • New learnings?

  • Useful file formats

  • Resources sufficient to data size

  • Data-chunking as technique if not enough RAM

  • Is Xarray/Polars/Dask useful for you?

Keypoints

  • Allocate more RAM by asking for

    • Several cores

    • Nodes will more RAM

    • Check job memory usage with sacct or sstat. Check you documentation!

  • File formats

    • No format fits all requirements

    • HDF5 and NetCDF good for Big data since it allows loading parts of the file into memory

  • Store temporary data in local scratch ($SNIC_TMP).

  • Packages
    • xarray

      • can deal with 3D-data and higher dimensions

    • Dask

      • uses lazy execution

      • Only use for processing very large amount of data

    • Chunking: Data source → Format choice → Load/Chunk → Process → Write

See also

ENCCS