1. Upload local files to fabric nodes
Searching...

Upload local files to fabric nodes

Home › Forums › FABRIC General Questions and Discussion › Upload local files to fabric nodes

Log In Register Lost Password
Viewing 8 posts - 1 through 8 (of 8 total)
  • Author
    Posts
  • April 12, 2022 at 4:00 pm #1626
    Xusheng Ai
    Participant

      Hello,

      As we plan to create a Genome data lake, we are testing the transfer speed between fabric nodes with iperf command. Firstly, we use truncate command to create non-meaning file with specific size, for example, 1 MB, 10 MB and etc. Then, we want to test the transfer with real Genome data. I download the Genome data on local desk, but I was wondering if it was possible to upload this data to the fabric node from local machine.

      Thanks,
      Best Regards,
      Xusheng

      April 13, 2022 at 8:08 am #1629
      Paul Ruth
      Keymaster

        The way to upload a file is to use a file transfer tool that uses ssh and can hop through the bastion host.  The most common of these is scp but there are others that might have better performance.

        FABLib includes an easy way to upload any file using paramiko. Any node object has a method called upload_file. This method will copy a file from the local host (i.e. your laptop or JupyterHub environment) to the remote node. It will put the file on the remote host in the location specified. It operates as the default user so there may be issues related to permission where you can write the file.

        Example:
        myNode.upload_file('/path/to/local/file', '/path/to/remote/file')

        More information about the FABLib upload method (and other functionality) can be found here: https://learn.fabric-testbed.net/docs/fablib/fablib.html#node.Node.upload_file

        Paul

        • This reply was modified 2 years, 7 months ago by Paul Ruth.
        • This reply was modified 2 years, 7 months ago by Paul Ruth.
        April 13, 2022 at 8:18 am #1633
        Xusheng Ai
        Participant

          Thank you so much for the information!

          April 13, 2022 at 8:48 am #1634
          Xusheng Ai
          Participant

            If I want to upload file from JupyterHub environment, what the local path should I use?

            April 13, 2022 at 8:56 am #1635
            Paul Ruth
            Keymaster

              The path should be to wherever the file you want to upload is located.

              For example, if you had a file in the base directory of your JupyterHub environment and you wanted to upload it to the home directory of the VM, you might use something like the following:

              mynode.upload_file('/home/fabric/work/myfile.txt', 'myfile.txt')

              April 13, 2022 at 9:56 am #1636
              Xusheng Ai
              Participant

                When I tried to upload a test.txt file from JupyterHub environment to node1, I got an error : Exception: cannot unpack non-iterable SFTPAttributes object

                Here is the script I run:

                try:
    node1 = slice.get_node(name = node1_name)
    stdout, stderr = node1.upload_file('/home/fabric/work/test.txt', 'test.txt')

except Exception as e:
    print(f"Exception: {e}")
                April 13, 2022 at 10:01 am #1637
                Paul Ruth
                Keymaster

                  Your code is trying to unpack the return value of upload file to the tuple (stdout,sdterr). The file upload is not a shell command and does not have a stdout/stderr.

                  Try setting the return value to a single variable. I think its a single string that shows the file attributes. The error is because you are trying to unpack that string into a tuple.

                  April 13, 2022 at 10:11 am #1638
                  Xusheng Ai
                  Participant

                    It works, thank you so much for the help!

                  • Author
                    Posts
                  Viewing 8 posts - 1 through 8 (of 8 total)
                  • You must be logged in to reply to this topic.