Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]
### Added
### Added
- Add Rayon global thread pool control via `FEOS_MAX_THREADS` and `set_num_threads()`/ `get_num_threads()` to Python. [#346](https://github.com/feos-org/feos/pull/346)
- Added DIPPR107 parameterization for ideal gas heat capacities of Burkhardt et al. [#344](https://github.com/feos-org/feos/pull/344)

## [0.9.4] - 2026-03-09
Expand Down
3 changes: 2 additions & 1 deletion docs/api/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,5 @@ All functions and classes in FeOS are exported at the package root. Here, they a
eos
dft
ad
```
thread_pool
```
38 changes: 38 additions & 0 deletions docs/api/thread_pool.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Global thread pool

Several functions in `feos` use [Rayon](https://github.com/rayon-rs/rayon) for parallelism.
By default, Rayon uses all logical CPUs available on your machine, which is usually what you want when working on your local machine.
In other environments, for example HPC clusters, you may want to limit the number of threads to match your job allocation.

There are three ways to configure this, in order of priority:

- `FEOS_MAX_THREADS` environment variable: for HPC or "script" environments, defined before launching Python
- `feos.set_num_threads()`: for interactive use, at the top of a script or notebook
- Do nothing: local machines where using all cores is fine

You can get the number of threads configured via `feos.get_num_threads()`.

## Important
- The thread pool can only be configured **once** per Python process.
- Whichever method runs first wins. Any later attempt to change it will have no effect and a warning will be emitted.
- Calling `get_num_threads` without setting `FEOS_MAX_THREADS` or `set_num_threads` will initialze the thread pool with the default (all logical CPUs).
- To test the different behaviour in a notebook, you have to restart the kernel and start from the respective cell you want to test.

## Example Usage

```python
import feos

feos.set_num_threads(4)
print(f"Active threads: {feos.get_num_threads()}")
```

```{eval-rst}
.. currentmodule:: feos

.. autosummary::
:toctree: generated/

set_num_threads
get_num_threads
```
314 changes: 314 additions & 0 deletions examples/managing_threads.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,314 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "b9685fcb",
"metadata": {},
"source": [
"# Controlling the Thread Pool in `feos`\n",
"\n",
"Several functions in `feos` use [Rayon](https://github.com/rayon-rs/rayon) for parallelism.\n",
"By default, Rayon uses all logical CPUs available on your machine, which is usually what you want when working on your local machine. \n",
"In other environments, for example HPC clusters, you may want to limit the number of threads to match your job allocation.\n",
"\n",
"There are three ways to configure this, in order of priority:\n",
"\n",
"- `FEOS_MAX_THREADS` environment variable: for HPC or \"script\" environments, defined before launching Python\n",
"- `feos.set_num_threads()`: for interactive use, at the top of a script or notebook\n",
"- Do nothing: local machines where using all cores is fine\n",
"\n",
"You can get the number of threads configured via `feos.get_num_threads()`.\n",
"\n",
"## Important\n",
"- The thread pool can only be configured **once** per Python process.\n",
"- Whichever method runs first wins. Any later attempt to change it will have no effect and a warning will be emitted.\n",
"- Calling `get_num_threads` without setting `FEOS_MAX_THREADS` or `set_num_threads` will initialze the thread pool with the default (all logical CPUs).\n",
"- To test the different behaviour in this notebook, restart the kernel and start from the respective cell you want to test."
]
},
{
"cell_type": "markdown",
"id": "34e0be01",
"metadata": {},
"source": [
"## Method 1: Environment variable (recommended for HPC)\n",
"\n",
"Set `FEOS_MAX_THREADS` **before** starting Python or launching your Jupyter kernel.\n",
"The thread pool is initialized automatically when `feos` is imported.\n",
"\n",
"In a Slurm job script:\n",
"\n",
"```bash\n",
"#!/bin/bash\n",
"#SBATCH --cpus-per-task=8\n",
"\n",
"export FEOS_MAX_THREADS=$SLURM_CPUS_PER_TASK\n",
"python my_script.py\n",
"```\n",
"\n",
"Or in a terminal before starting Jupyter:\n",
"\n",
"```bash\n",
"export FEOS_MAX_THREADS=4\n",
"jupyter lab\n",
"```\n",
"\n",
"You can verify it was picked up after importing:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "3e161b43",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Active threads: 10\n"
]
}
],
"source": [
"import feos\n",
"\n",
"# If FEOS_MAX_THREADS was set before starting Python, the pool\n",
"# was already configured at import time.\n",
"print(f\"Active threads: {feos.get_num_threads()}\")"
]
},
{
"cell_type": "markdown",
"id": "b60da8bf",
"metadata": {},
"source": [
"## Method 2: `set_num_threads()` (interactive use)\n",
"\n",
"If you did not set the environment variable, you can configure the thread pool\n",
"programmatically. This must be done **before any parallel computation is triggered**.\n",
"\n",
"In a notebook or script, call it immediately after importing `feos`: (if the code below emits a warning, restart the kernel and run the code below as first cell)"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "d5fce92f",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Active threads: 4\n"
]
}
],
"source": [
"import feos\n",
"\n",
"feos.set_num_threads(4)\n",
"\n",
"print(f\"Active threads: {feos.get_num_threads()}\")"
]
},
{
"cell_type": "markdown",
"id": "b8a862cd",
"metadata": {},
"source": [
"You can also read the thread count from `SLURM_CPUS_PER_TASK` manually if you prefer\n",
"to keep configuration in Python rather than in your shell environment:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "a8e08786",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Active threads: 4\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/var/folders/3s/t93ws1md04qdbbq5d1jdz8640000gn/T/ipykernel_9962/4108200440.py:5: UserWarning: set_num_threads(10) without effect: The thread pool was already initialized with 4 thread(s) Call set_num_threads() before any parallel work or set FEOS_MAX_THREADS before starting Python.\n",
" feos.set_num_threads(n_threads)\n"
]
}
],
"source": [
"import os\n",
"import feos\n",
"\n",
"n_threads = int(os.environ.get(\"SLURM_CPUS_PER_TASK\", os.cpu_count()))\n",
"feos.set_num_threads(n_threads)\n",
"\n",
"print(f\"Active threads: {feos.get_num_threads()}\")"
]
},
{
"cell_type": "markdown",
"id": "9c891bdc",
"metadata": {},
"source": [
"## Method 3: Do nothing (Rayon default)\n",
"\n",
"If you neither set `FEOS_MAX_THREADS` nor call `set_num_threads()`, Rayon will initialize the thread pool lazily the first time a parallel function is called, using all available logical CPUs. This is usually the right choice on a local workstation.\n",
"Note that calling `get_num_threads` will set the threads to the default."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "61c0c96d",
"metadata": {},
"outputs": [],
"source": [
"import feos\n",
"\n",
"# No configuration — Rayon will use all logical CPUs.\n",
"# get_num_threads() triggers lazy initialization if not already done.\n",
"print(f\"Active threads: {feos.get_num_threads()}\")"
]
},
{
"cell_type": "markdown",
"id": "142bc70d",
"metadata": {},
"source": [
"## What happens if you call `set_num_threads()` too late?\n",
"\n",
"If the pool is already initialized — because `FEOS_MAX_THREADS` was set, or because\n",
"a parallel function (or `get_num_threads()`) has already run — `set_num_threads()`\n",
"has no effect and emits a warning:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cba243c9",
"metadata": {},
"outputs": [],
"source": [
"import feos\n",
"\n",
"print(feos.get_num_threads()) # triggers lazy initialization\n",
"\n",
"feos.set_num_threads(2) # UserWarning: had no effect"
]
},
{
"cell_type": "markdown",
"id": "5ba53f01",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"```\n",
" import feos\n",
" │\n",
" FEOS_MAX_THREADS set?\n",
" ┌──────┴───────┐\n",
" Yes No\n",
" │ │\n",
" Pool initialized set_num_threads() called?\n",
" with env var value ┌──────┴───────┐\n",
" Yes No\n",
" │ │\n",
" Pool initialized Pool initialized lazily\n",
" with given value on first parallel call\n",
" (all logical CPUs)\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "fe867ce1-10f6-43b4-8382-b9f5bc9b387d",
"metadata": {},
"source": [
"## Example\n",
"\n",
"The following example calculates vapor pressures and derivatives w.r.t. the model's parameters in parallel.\n",
"Set the number of threads to check the impact on calculation time (restart kernel as before to change the threads)."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "3ee76e3a-6727-45c4-8ca1-58f6c3b27231",
"metadata": {},
"outputs": [],
"source": [
"import feos\n",
"import numpy as np\n",
"import timeit\n",
"\n",
"# modify the number of threads and rerun the cell below to see impact\n",
"feos.set_num_threads(0) \n",
"\n",
"n = 1_000_000\n",
"fit_params = [\"m\", \"sigma\", \"epsilon_k\"]\n",
"\n",
"# order: m, sigma, epsilon_k, mu\n",
"parameters = np.array([[1.5, 3.4, 230.0, 2.3]] * n)\n",
"temperature = np.expand_dims(np.linspace(250.0, 400.0, n), 1)\n",
"eos = feos.EquationOfStateAD.PcSaftNonAssoc"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "4315f002-ce50-4e01-a2aa-49f28eebd84d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Mean of 5 runs (1000000 VLEs each): 0.559s using 10 thread(s)\n"
]
}
],
"source": [
"n_runs = 5\n",
"n_threads = feos.get_num_threads()\n",
"elapsed = np.mean(timeit.repeat(\n",
" lambda: feos.vapor_pressure_derivatives(eos, fit_params, parameters, temperature),\n",
" number=1,\n",
" repeat=n_runs\n",
"))\n",
"\n",
"print(f\"Mean of {n_runs} runs ({n} VLEs each): {elapsed:.3f}s using {n_threads} thread(s)\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Loading
Loading