Skip to content

Known Issues

This page provides information about current known issues that may affect users of BriCS services and facilities. If you have encountered an issue that is not listed on this page, or you are unable to resolve a known issue using a provided workaround, please get in touch with the BriCS team.

BriCS Services and Facilities

Jobs stuck in completing (CG) state

Last updated: 2026-04-14

Services affected

Isambard-AI Phase 1, Isambard-AI Phase 2

Description

Jobs may occasionally become stuck in the completing (CG) state in Slurm. When this occurs, attempts to cancel the affected job using scancel will not successfully terminate it, and the job will remain in the queue. Node resources allocated to these jobs will also remain occupied until the job is cleaned up. Affected jobs are not automatically killed in a timely manner as would normally be expected.

This is a known issue that the BriCS team is actively working to resolve. No action is required on the part of the user.

Workaround

There is no workaround. Jobs affected by this issue are safe to treat as complete. They will be automatically cleaned up, though this may take some time.

Missing LOCALDIR and TMPDIR for Jupyter sessions

Last updated: 2025-11-07

Services affected

Isambard-AI Phase 2 JupyterHub

Description

The /run/user/$UID directory is not present in interactive JupyterLab sessions started from JupyterHub (https://apps.isambard.ac.uk/aip2-jupyter/). On Isambard-AI Phase 2 compute nodes the LOCALDIR and TMPDIR environment variables point to this directory via a symbolic link. As a consequence, attempts to access LOCALDIR or TMPDIR from within the JupyterLab session will fail.

Workaround

If the LOCALDIR or TMPDIR environment variables are required for your workflow, manually set these to an alternative location (e.g. a subdirectory in the per-user scratch space). Note that using a location on the shared parallel filesystem may cause issues if LOCALDIR or TMPDIR is used for rootless container storage.

Alternatively submit workloads which require node-local LOCALDIR or TMPDIR from an SSH session using sbatch or srun (only jobs submitted by JupyterHub on the web are affected).

Use of environment variables in apptainer.

Last update: 2025-10-16

Services affected

Isambard 3 MACS, Isambard-AI Phase 1, Isambard-AI Phase 2

Description

Slurm, by default, inherits environment variables from the submission host (e.g. the login node). This can pass special environment variables such as DBUS_SESSION_BUS_ADDRESS and XDG_RUNTIME_DIR. These variables can trigger different logic in applications and cause unexpected issues such as in apptainer.

Workaround

It is recommended to unset these variables if you experience issues with apptainer.

OpenSSH post-quantum key exchange warning

Last updated: 2026-02-16

Services affected

Login services

Description

When using ssh to log in to BriCS facilities, a user may be using a recent version of OpenSSH that displays the following:

** WARNING: connection is not using a post-quantum key exchange algorithm.
** This session may be vulnerable to "store now, decrypt later" attacks.
** The server may need to be upgraded. See https://openssh.com/pq.html
Workaround

No current workaround. ssh servers will soon be upgraded to take advantage of the newly recommended post-quantum key exchange algorithms.

Third Party

Issues with third party services

Issues with products not managed by BriCS are listed below. Where possible we will provide workarounds and report issues to the service provider, but will usually need to rely on the third party service provider to resolve the underlying issue.

Multi-node Singularity/Apptainer: PyTorch NGC container image GLIBC mismatch

Last updated: 2025-07-25

Services affected

Isambard-AI Phase 2

Description

There are GLIBC mismatches when using multi-node containers. When using Singularity/Apptainer with the brics/apptainer-multi-node module, Podman-HPC with --openmpi-pmi2, the system GLIBC is incompatible with some versions of the PyTorch NGC container.

Workaround

Use one of the following versions which are known to work correctly:

  • nvcr.io/nvidia/pytorch:24.12-py3
  • nvcr.io/nvidia/pytorch:25.01-py3
  • nvcr.io/nvidia/pytorch:25.02-py3
  • nvcr.io/nvidia/pytorch:25.03-py3
  • nvcr.io/nvidia/pytorch:25.04-py3
  • nvcr.io/nvidia/pytorch:25.05-py3

Slurm bug when using overlap and requesting GPUs

Last updated: 2026-02-13

Services affected

Isambard 3 MACS, Isambard-AI Phase 1, Isambard-AI Phase 2

Description

When requesting X number of GPUs across multiple nodes in Slurm and overlapping an existing job with --overlap, it will not function as expected where the original job used --gpus=X across N nodes with --nodes=N. This may produce an error such as Insufficient GRES available in allocation. This is not encountered when using --gres=gpu:X which specifies the GRES per node.

Workaround

It is recommended to specify --gpus-per-node and --nodes in the original job instead. This is fixed upstream in Slurm.

Slurm incorrectly reports available GPUs in overlapped jobs

Last updated: 2026-05-19

Services affected

Isambard-AI Phase 1, Isambard-AI Phase 2

Description

When requesting to overlap a job with an existing Slurm allocation the available of GPUs may be incorrect if 2 or more GPUs are requested per node and using the option --cpus-per-task in the original job. The original Slurm should still see the correct number of GPUs. This is usually seen when running nvidia-smi or checking CUDA_VISIBLE_DEVICES.

Workaround

If using all GPUs on a node --exclusive can be used. Otherwise use --cpus-per-gpu and remove --cpus-per-task in the original Slurm job.

Fortran library changes can cause run-time errors

Last updated: 2026-07-02

Services affected

Isambard 3

Description

When running code compiled with gfortran on an older operating system, a newer gfortran library can cause runtime errors having detected bugs in the code with aspects such as a missing comma between edit descriptors. Example error may be Fortran runtime error: Missing comma.

Workaround

Recompile the code with following gfortran option -std=legacy or fix the bugs in the code.

Archived

For information on previous known issues that are now resolved, see the Archived Issues page.