File Transfer with scp

Using the SCP command

For MacOS, Linux, Windows 10 and Windows 11 users, file transferring between your personal computer and the HCC supercomputers can be achieved through the command scp which stands for secure copy. This method is ideal for quick transfer of smaller files. For large volume transfers, we recommend the using [Globus] or an SCP client such as WinSCP for Windows or CyberDuck for Mac/Linux.

Just like the cp copy command, the scp command requires two arguments, the path to source file(s) and the path to the target location. Since one or more of these locations are remote, you will need to specify the username and host for those.

$ scp <username>@<host>:<path_to_files> <username>@<host>:<path_to_files>

For the local location, you do not need to specify the username or host. When transferring to and from your local computer, the scp command should be ran on your computer, NOT from HCC clusters.

Uploading a file to Swan

Here is an example of file transfer to and from the Swan cluster.

To upload the file data.csv in your current directory to your $WORK directory on the Swan cluster, you would use the command:

$ scp ./data.csv <user_name>@swan.unl.edu:/work/<group_name>/<user_name>

where <user_name> and <group_name> are replaced with your user name and your group name.

Downloading a file from Swan

To download the file data.csv from your $WORK directory on the Swan cluster to your current directory, you would use the command:

$ scp <user_name>@swan.unl.edu:/work/<group_name>/<user_name>/data.csv ./

Potential incompatibility with recent versions of scp

Note

Recent versions of scp command when used with shell variable or pathname expanded source/target arguments, along with certain recursive copy requests, may return unexpected errors compared with older versions of the command.

Use the -O option to restore the prior behavior and work around the issue.

Beginning with OpenSSH release 8.8, the scp client defaults to using the SFTP protocol over the legacy SCP protocol. This change in protocol removes the user's shell from completing environment variable expansion or wildcard pathname matching on the path component being serviced by the remote server side of the transfer, as used in the SCP protocol. This may cause scp transfers to return errors that worked with prior versions of scp. Some example error messages are listed below.

Example error messages

# Recursive copy incompatibility example.
# <source_dir> is missing from the target path resulting in error:
$ scp -r <source_dir> <user_name>@swan.unl.edu:/work/<group_name>/<user_name>/
scp: realpath /work/<group_name>/<user_name>/<source_dir>: No such file
scp: upload "/work/<group_name>/<user_name>/<source_dir>": path canonicalization failed
scp: failed to upload directory <source_dir> to /work/<group_name>/<user_name>/

# Shell environment expansion incompatibility example.
# $WORK is treated as a literal string and not expanded by the remote shell:
$ scp <user_name>@swan.unl.edu:'$WORK/path/to/file' .
scp: $WORK/path/to/file: No such file or directory

To restore the prior shell expansion behavior on the remote server side of the transfer, add the -O flag in your scp invocation to use the SCP protocol.

# Creates <source_dir> and its contents entirely at the target path:
$ scp -O -r <source_dir> <user_name>@swan.unl.edu:/work/<group_name>/<user_name>/
# Uses the remote shell on swan.unl.edu to expand $WORK to the string
# that is <user_name>'s full "/work/<group_name>/<user_name>" path.
# Note the single quotes to keep the local shell from where scp is invoked
# from attempting to expand the $WORK variable:
$ scp -O <user_name>@swan.unl.edu:'$WORK/path/to/file' .

Details of the change are available at the OpenSSH release 8.8 Future deprecation notice section.