Skip to content

Transfer files between your local machine and BriCS facilities

Prerequisites

To work through this guide you must

  • Be a member of a project in the BriCS portal
  • Have followed the instructions in Getting Started to set up SSH login and create a valid SSH certificate for authenticating to BriCS facilities (using the clifton command line tool)
  • PROJECT and FACILITY within this page correspond to the values that appear in SSH host names generated by clifton, as described in the Login guide

Having setup ssh access as described in Getting Started, you can use scp or rsync to transfer files over SSH.

rsync on macOS

The version of rsync bundled with macOS is very old and can be significantly slower than current releases. If you are on macOS, we recommend installing a newer version before transferring files:

  • Homebrew: brew install rsync (see brew.sh)
  • MacPorts: sudo port install rsync (see macports.org)

For example to copy remote_file.txt from the remote login node to your local machine, the following can be used:

scp PROJECT.FACILITY.isambard:remote_file.txt .

rsync -avz PROJECT.FACILITY.isambard:remote_file.txt .

To copy local_file.txt from your local machine to the remote login node, the following can be used:

scp local_file.txt PROJECT.FACILITY.isambard:

rsync -avz local_file.txt PROJECT.FACILITY.isambard:

To copy local_file.txt from your local machine to another storage space on the BriCS facility e.g. your project public shared storage, the following can be used:

scp local_file.txt PROJECT.FACILITY.isambard:/projects/public/PROJECT

rsync -avz local_file.txt PROJECT.FACILITY.isambard:/projects/public/PROJECT

To copy a directory recursively, we use the -r flag with scp, and the -a flag with rsync. For example:

scp -r local_dir PROJECT.FACILITY.isambard:

rsync -avz local_dir PROJECT.FACILITY.isambard:

Transferring large number of files

Transferring a large number of small files can lead to inefficiencies as each file requires to be accessed, transferred and checked. It can be more efficient using the tar command to package up a large number of files into a single file to transfer.

For example to transfer using tar from a local machine to BriCS facility:

$ tar c local_dir | ssh PROJECT.FACILITY.isambard 'cat > remote_file.tar'

The cat > remote_file.tar command can be replaced with tar x to extract the archive automatically on the remote machine.

Alternatively to transfer using tar from BriCS facility to a local machine:

$ ssh PROJECT.FACILITY.isambard 'tar c remote_dir' > local_file.tar

The > local_file.tar can be replaced with | tar x to pipe directly to tar to extract contents.

Finally, to extract a tar file local_file.tar with tar use tar xf local_file.tar