Skip to content

Latest commit

 

History

History
176 lines (142 loc) · 7.44 KB

README.md

File metadata and controls

176 lines (142 loc) · 7.44 KB

Build Status

Python Scripting with Appose

This library provides a ScriptLanguage plugin for the SciJava Common platform that runs scripts in CPython (not Jython!), each in its own Python environment, using Appose.

Scripts are ordinary SciJava scripts: they declare script parameters and can be run from the Script Editor or menus of ImageJ2/Fiji like any other script.

Writing a script

#!appose-python
# /// script
# requires-python = ">=3.12"
# dependencies = ["appose", "scipy"]
# ///

#@ Img image
#@ double sigma
#@output Img blurred

from scipy.ndimage import gaussian_filter

print(f"Blurring image of shape {image.shape}")
blurred = gaussian_filter(image, sigma)
  • The #!appose-python line selects this script language, rather than another language handling .py files, such as Jython.
  • The # /// script block declares the script's Python environment; see below.
  • The environment must include the appose Python package, plus numpy if the script uses images.

See the StarDist_cellcast.py template for a complete example.

Environments

A script declares its environment in one of two ways.

Inline, with a PEP 723 metadata block, as above. requires-python selects the Python version, and dependencies lists packages from PyPI. The [tool.pixi.*] extensions of pixi run --script are supported too, so conda packages are available, e.g. to get CUDA libraries from conda-forge:

# /// script
# requires-python = ">=3.12"
# dependencies = ["appose"]
#
# [tool.pixi.dependencies]
# pytorch-gpu = "*"
#
# [tool.pixi.system-requirements]
# cuda = "12"
# ///

If [tool.pixi.workspace] does not say otherwise, packages come from conda-forge, for the platform the script runs on. Inline environments are the way to go for scripts that must stand alone, such as templates.

In a file, named by the env attribute of the #@script directive:

#@script(env="myenv.toml")

The file is resolved relative to the script's location. Any format Appose supports works: pixi.toml, environment.yml, requirements.txt or pyproject.toml. An optional scheme attribute (e.g. scheme="pixi.toml") sets the format explicitly, which is useful when the file name does not reveal it. Scripts that share a file share an environment.

A script cannot do both.

The environment is built the first time a script needs it, which may take a while. Afterward, it is reused, both within the running application and across restarts. Environments are stored in the Appose environments directory (~/.local/share/appose by default), named after the file declaring them: the environment file, or the script itself for inline environments. Editing the declaration causes the environment to be updated on the next run.

Builds appear in the application's task list, with their progress, so a long first build does not look like a hang.

Python workers

Each environment gets one Python worker process, which stays alive between script runs. Modules a script imports stay imported, so a second run of a script that imports e.g. PyTorch starts quickly. Each run still gets a fresh namespace: variables from one run are not visible to the next.

To keep something expensive, such as a loaded model, across runs, hand it to task.export:

if "model" not in globals():
    model = load_model()
    task.export(model=model)

Runs on the same worker execute one at a time; since each environment currently gets a single worker, so do runs on the same environment, while runs on different environments proceed in parallel. Each run appears in the application's task list, where it can be canceled. A script can notice cancelation by checking task.cancel_requested; if it has not stopped a few seconds after being canceled, its worker process is stopped, and the next run starts a new one. The worker, and whatever memory (including GPU memory) it holds, is otherwise released when the application exits.

Inputs and outputs

  • Images (anything convertible to an Appose NDArray, such as an ImgLib2 Img) are passed via shared memory and appear in Python as NumPy arrays. Axes are in NumPy's usual order, which is the reverse of ImgLib2's: an image with X×Y dimensions 512×384 has NumPy shape (384, 512).
  • Numbers, strings, booleans, lists and maps are passed as the corresponding Python values; files are passed as path strings.
  • Outputs are read from the Python variables of the same names after the script finishes. NumPy arrays are converted back to images; NumPy scalars become plain numbers. Outputs the script never assigns are left empty.
  • If the script declares no outputs and ends with an expression, its value becomes the script's result.

Text printed to stdout and stderr appears in the Script Editor's output panes. Errors are reported with a Python traceback that refers to the script's own line numbers.

Code completion

The Script Editor completes code in these scripts with a Python language server, scijava-lsp-ty's: ty. This plugin's ApposePythonDialect shows it each script as plain Python, and says where the script runs (so any other client of scijava-lsp's LanguageServerService gets the same). It completes names in scope, the script's #@ parameters, task, builtins and keywords, and members of objects and modules; and it gives function signatures (with parameter names and types), documentation, hover and syntax errors.

The server runs in a small environment of its own (code-lsp-ty: Python, NumPy and ty), which is built in the background the first time you complete code, and takes a while only then; until it is, there are no completions. The server talks to the editor via the Language Server Protocol (see scijava-lsp); it sees the script as plain Python, with its #@ parameters declared. One server runs per script environment; servers unused for 10 minutes stop. Their error output goes to code-lsp-ty-server.log, next to Appose's environments.

Once a script's environment is built (in this session or an earlier one), the server analyzes the script against it, so that the packages installed there are known too. An environment not built yet, e.g. because you just added a dependency, is built in the background once the script's environment declaration has stayed the same for a few seconds; so its packages are known before you run the script, and the next run need not build it. This covers every kind of declaration: PyPI and conda packages, and environment files. A script running in the environment is waited for first, so an environment is never rebuilt beneath a running script; building replaces the environment's worker, as the script's next run would. Script parameters are known by the Python types they have at run time, e.g. images as NumPy arrays.

Typing never waits for the server: if its answer for a word is not ready within a moment (the first answer for a package can take a few seconds), it is shown once ready. Questions that are no longer of interest (e.g. as you type on) are cancelled.