Upload local files to fabric nodes

[Xusheng Ai]

Hello,

As we plan to create a Genome data lake, we are testing the transfer speed between fabric nodes with iperf command. Firstly, we use truncate command to create non-meaning file with specific size, for example, 1 MB, 10 MB and etc. Then, we want to test the transfer with real Genome data. I download the Genome data on local desk, but I was wondering if it was possible to upload this data to the fabric node from local machine.

Thanks,
Best Regards,
Xusheng

[Paul Ruth]

The way to upload a file is to use a file transfer tool that uses ssh and can hop through the bastion host.  The most common of these is scp but there are others that might have better performance.

FABLib includes an easy way to upload any file using paramiko. Any node object has a method called upload_file. This method will copy a file from the local host (i.e. your laptop or JupyterHub environment) to the remote node. It will put the file on the remote host in the location specified. It operates as the default user so there may be issues related to permission where you can write the file.

Example:
myNode.upload_file('/path/to/local/file', '/path/to/remote/file')

More information about the FABLib upload method (and other functionality) can be found here: https://learn.fabric-testbed.net/docs/fablib/fablib.html#node.Node.upload_file

Paul

• This reply was modified 2 years, 7 months ago by Paul Ruth.
• This reply was modified 2 years, 7 months ago by Paul Ruth.

[Xusheng Ai]

Thank you so much for the information!

[Xusheng Ai]

If I want to upload file from JupyterHub environment, what the local path should I use?

[Paul Ruth]

The path should be to wherever the file you want to upload is located.

For example, if you had a file in the base directory of your JupyterHub environment and you wanted to upload it to the home directory of the VM, you might use something like the following:

mynode.upload_file('/home/fabric/work/myfile.txt', 'myfile.txt')

[Xusheng Ai]

When I tried to upload a test.txt file from JupyterHub environment to node1, I got an error : Exception: cannot unpack non-iterable SFTPAttributes object

Here is the script I run:

try:
    node1 = slice.get_node(name = node1_name)
    stdout, stderr = node1.upload_file('/home/fabric/work/test.txt', 'test.txt')

except Exception as e:
    print(f"Exception: {e}")

[Paul Ruth]

Your code is trying to unpack the return value of upload file to the tuple (stdout,sdterr). The file upload is not a shell command and does not have a stdout/stderr.

Try setting the return value to a single variable. I think its a single string that shows the file attributes. The error is because you are trying to unpack that string into a tuple.

[Xusheng Ai]

It works, thank you so much for the help!

[Author]

Posts