Skip to content

File Transfer

Prerequisites

To work through these guides you must

  • Be a member of a project in the BriCS portal
  • Have followed the instructions in Getting Started to set up SSH login and create a valid SSH certificate for authenticating to BriCS facilities (using the clifton command line tool)
  • PROJECT and FACILITY within this page correspond to the values that appear in SSH host names generated by clifton, as described in the Login guide

This guide outlines how to:

If you wish to transfer data between different BriCS facilities (noting that Isambard-AI phase 1 and phase 2 are different facilities), please copy via an external location. BriCS staff do not copy or move data on behalf of users, and bespoke transfers cannot be arranged by exception. All the transfers described on this page are ones you run yourself.

Transferring files between your local machine and BriCS facilities

Having setup ssh access as described in Getting Started, you can use scp or rsync to transfer files over SSH protocol.

rsync on macOS

The version of rsync bundled with macOS is very old and can be significantly slower than current releases. If you are on macOS, we recommend installing a newer version before transferring files:

  • Homebrew: brew install rsync (see brew.sh)
  • MacPorts: sudo port install rsync (see macports.org)

For example to copy remote_file.txt from the remote login node to your local machine, the following can be used:

scp PROJECT.FACILITY.isambard:remote_file.txt .

rsync -avz PROJECT.FACILITY.isambard:remote_file.txt .

To copy local_file.txt from your local machine to the remote login node, the following can be used:

scp local_file.txt PROJECT.FACILITY.isambard:

rsync -avz local_file.txt PROJECT.FACILITY.isambard:

To copy local_file.txt from your local machine to another storage space on the BriCS facility e.g. your project public shared storage, the following can be used:

scp local_file.txt PROJECT.FACILITY.isambard:/projects/public/PROJECT

rsync -avz local_file.txt PROJECT.FACILITY.isambard:/projects/public/PROJECT

To copy a directory recursively, we use the -r flag with scp, and the -a flag with rsync. For example:

scp -r local_dir PROJECT.FACILITY.isambard:

rsync -avz local_dir PROJECT.FACILITY.isambard:

Transferring large number of files

Transferring a large number of small files can lead to inefficiencies as each file requires to be accessed, transferred and checked. It can be more efficient using the tar command to package up a large number of files into a single file to transfer.

For example to transfer using tar from a local machine to BriCS facility:

$ tar c local_dir | ssh PROJECT.FACILITY.isambard 'cat > remote_file.tar'

The cat > remote_file.tar command can be replaced with tar x to extract the archive automatically on the remote machine.

Alternatively to transfer using tar from BriCS facility to a local machine:

$ ssh PROJECT.FACILITY.isambard 'tar c remote_dir' > local_file.tar

The > local_file.tar can be replaced with | tar x to pipe directly to tar to extract contents.

Finally, to extract a tar file local_file.tar with tar use tar xf local_file.tar

Transferring files between projects on the same BriCS facility

Seek permission from your PI

When transferring data between projects, care should be taken not to breach privacy or licensing. Please consult with the PI of the project before transferring data. The PI is legally responsible for any breach.

To transfer files from one project to another on the same facility, we outline the following methods:

Using public shared storage

You can use cp to copy files to and from your project public shared storage. E.g. to copy a directory recursively to your project public shared storage:

cp -r dir /projects/public/PROJECT

Project Public Shared Storage

/projects/public/PROJECT is configured to be writeable by members of the project, and readable by all users of the facility.

Using scp/rsync and clifton

You will need to install and run clifton from your source project account. First, you should generate a ssh-key pair if you have not already done so, then follow the Clifton "Install" and "Authenticate" instructions.

Once you have authenticated using Clifton and ran clifton ssh-config write, you'll need to make a small change to the Clifton SSH config file, commenting out the ProxyJump directive under the relevant Host in ~/.ssh/config_clifton.

You can then use scp or rsync to transfer files between projects on the same facility. E.g. to transfer a directory recursively from your source project account to your destination project PROJECT2:

scp -r dir PROJECT2.FACILITY.isambard:

rsync -avz dir PROJECT2.FACILITY.isambard:

Transferring large amounts of data as a job

The login nodes are a shared, resource-limited interface to the facility and are not a free transfer resource. Each login node session on Isambard-AI is limited to the equivalent of one CPU core and 4 GiB of memory, so a large copy started on a login node is likely to be stopped by the out of memory killer long before it finishes. See the FAQ entry on login node resource limits for the current values.

For anything larger than a handful of files, submit the transfer to the compute nodes with the Slurm workload manager instead. Request a single task on a single node, as the transfer is limited by storage and network throughput rather than by the compute node itself. On Isambard-AI, request one GPU, which allocates one complete GH200 Superchip and its memory.

#!/bin/bash

#SBATCH --job-name=project-transfer
#SBATCH --output=project-transfer.out
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --gpus=1
#SBATCH --time=06:00:00

rsync --archive --verbose --partial --human-readable dir PROJECT2.FACILITY.isambard:

#!/bin/bash

#SBATCH --job-name=project-transfer
#SBATCH --output=project-transfer.out
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --time=06:00:00

rsync --archive --verbose --partial --human-readable dir PROJECT2.FACILITY.isambard:

Submit the script with sbatch:

user.project@nid001040:~> sbatch project-transfer.sh
Submitted batch job 19159

The transfer then runs without your session, so you can log out and check the job's output file once it has completed. Set --time generously enough for the whole copy to finish, remembering that the maximum job run time is 24 hours. Because rsync --partial keeps partially transferred files, re-running the same job will resume rather than start again if the transfer is interrupted or runs out of time.

Transfers can be run as a pull as well as a push

A transfer does not have to be started from the project that holds the data. You can run the job in the source project and push the data to the destination:

rsync --archive --verbose --partial --human-readable dir PROJECT2.FACILITY.isambard:

Or run the job in the destination project and pull the data from the source:

rsync --archive --verbose --partial --human-readable PROJECT1.FACILITY.isambard:dir .

Both directions work in the same way. clifton must be installed and authenticated in whichever project account you start the transfer from, as described above.

The direction matters when one of the two projects has expired. Only a project that is still active can submit jobs, so run the transfer from the active project and pull the data across.

Data movement consumes your compute allocation

A transfer submitted as a job consumes node hours in the same way as any other job, and this is expected. On Isambard-AI, a single GPU consumes 0.25 node hours per hour of wall clock time. See the accounting guide for details. The login nodes are not a cheaper alternative for large transfers, and using them as one degrades the service for every other user.

Plan data movement into your allocation, as jobs will not run once your project's node hours are exhausted.

Copy your data out before your project ends

Jobs cannot be submitted for a project once it has expired, so this method is not available to you during the 30-day grace period. Copying from a login node is then the only option left, and for a large volume of data that is slow and difficult.

Treat data movement as part of your project and copy your data out before the project end date, rather than leaving it to the grace period.

There is one exception. A copy into another project that is still active can be run as a job by that project, because the active project can still submit jobs even though the expired one cannot. Run the transfer from the active project and pull the data from the expired project, as described above.

Check your SSH certificate before submitting

The SSH certificates issued by clifton are valid for 12 hours. Authentication happens when the job starts, not when it is submitted, so a job that starts after the certificate has expired will fail immediately. Run clifton auth shortly before submitting, and allow for the time the job may spend queueing.

Graphical interfaces

There are number of graphical file transfer programs and we currently do not provide specific instructions for a particular program.

As an alternative, if you simply want to view and work with your remote files using a graphical interface (rather than transferring them to your local machine), consider using Visual Studio Code or Jupyter notebooks.