Transfer files between projects on the same BriCS facility

Prerequisites

To work through this guide you must

• Be a member of a project in the BriCS portal
• Have followed the instructions in Getting Started to set up SSH login and create a valid SSH certificate for authenticating to BriCS facilities (using the clifton command line tool)
• PROJECT and FACILITY within this page correspond to the values that appear in SSH host names generated by clifton, as described in the Login guide

Seek permission from your PI

When transferring data between projects, care should be taken not to breach privacy or licensing. Please consult with the PI of the project before transferring data. The PI is legally responsible for any breach.

To transfer files from one project to another on the same facility, we outline the following methods:

• Transfer via public shared storage
• Transfer using scp or rsync (requires clifton)

Using public shared storage

You can use cp to copy files to and from your project public shared storage. E.g. to copy a directory recursively to your project public shared storage:

cp -r dir /projects/public/PROJECT

Project Public Shared Storage

/projects/public/PROJECT is configured to be writeable by members of the project, and readable by all users of the facility.

Using scp/rsync and clifton

You will need to install and run clifton from your source project account. First, you should generate a ssh-key pair if you have not already done so, then follow the Clifton "Install" and "Authenticate" instructions.

Once you have authenticated using Clifton and ran clifton ssh-config write, you'll need to make a small change to the Clifton SSH config file, commenting out the ProxyJump directive under the relevant Host in ~/.ssh/config_clifton.

You can then use scp or rsync to transfer files between projects on the same facility. E.g. to transfer a directory recursively from your source project account to your destination project PROJECT2:

scprsync

scp -r dir PROJECT2.FACILITY.isambard:

rsync -avz dir PROJECT2.FACILITY.isambard:

Transferring large amounts of data as a job

The login nodes are a shared, resource-limited interface to the facility and are not a free transfer resource. Each login node session on Isambard-AI is limited to the equivalent of one CPU core and 4 GiB of memory, so a large copy started on a login node is likely to be stopped by the out of memory killer long before it finishes. See the FAQ entry on login node resource limits for the current values.

For anything larger than a handful of files, submit the transfer to the compute nodes with the Slurm workload manager instead. Request a single task on a single node, as the transfer is limited by storage and network throughput rather than by the compute node itself. On Isambard-AI, request one GPU, which allocates one complete GH200 Superchip and its memory.

Isambard-AIIsambard 3

#!/bin/bash

#SBATCH --job-name=project-transfer
#SBATCH --output=project-transfer.out
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --gpus=1
#SBATCH --time=06:00:00

rsync --archive --verbose --partial --human-readable dir PROJECT2.FACILITY.isambard:

#!/bin/bash

#SBATCH --job-name=project-transfer
#SBATCH --output=project-transfer.out
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --time=06:00:00

rsync --archive --verbose --partial --human-readable dir PROJECT2.FACILITY.isambard:

Submit the script with sbatch:

user.project@nid001040:~> sbatch project-transfer.sh
Submitted batch job 19159

The transfer then runs without your session, so you can log out and check the job's output file once it has completed. Set --time generously enough for the whole copy to finish, remembering that the maximum job run time is 24 hours. Because rsync --partial keeps partially transferred files, re-running the same job will resume rather than start again if the transfer is interrupted or runs out of time.

Transfers can be run as a pull as well as a push

A transfer does not have to be started from the project that holds the data. You can run the job in the source project and push the data to the destination:

rsync --archive --verbose --partial --human-readable dir PROJECT2.FACILITY.isambard:

Or run the job in the destination project and pull the data from the source:

rsync --archive --verbose --partial --human-readable PROJECT1.FACILITY.isambard:dir .

Both directions work in the same way. clifton must be installed and authenticated in whichever project account you start the transfer from, as described above.

The direction matters when one of the two projects has expired. Only a project that is still active can submit jobs, so run the transfer from the active project and pull the data across.

Data movement consumes your compute allocation

A transfer submitted as a job consumes node hours in the same way as any other job, and this is expected. On Isambard-AI, a single GPU consumes 0.25 node hours per hour of wall clock time. See the accounting guide for details. The login nodes are not a cheaper alternative for large transfers, and using them as one degrades the service for every other user.

Plan data movement into your allocation, as jobs will not run once your project's node hours are exhausted.

Copy your data out before your project ends

Jobs cannot be submitted for a project once it has expired, so this method is not available to you during the 30-day grace period. Copying from a login node is then the only option left, and for a large volume of data that is slow and difficult.

Treat data movement as part of your project and copy your data out before the project end date, rather than leaving it to the grace period.

There is one exception. A copy into another project that is still active can be run as a job by that project, because the active project can still submit jobs even though the expired one cannot. Run the transfer from the active project and pull the data from the expired project, as described above.

Check your SSH certificate before submitting

The SSH certificates issued by clifton are valid for 12 hours. Authentication happens when the job starts, not when it is submitted, so a job that starts after the certificate has expired will fail immediately. Run clifton auth shortly before submitting, and allow for the time the job may spend queueing.