Fablib threading/concurrency

[Timothy Middelkoop]

I have a long running task (install software) to do during setup and would like to do it on all nodes simultaneously.  I did not see any async/non-blocking execute function in the fabrictestbed_extensions/fablib library.  I tried to run the methods with python’s multi-process (massive fail) as well as the concurrent.futures.ThreadPoolExecutor (only completed one node).  Is the library thread/multiprocessing safe?  Any plans on it? Any plans on making async/non-blocking versions of node.execute?

Here is a snippit of my code

with concurrent.futures.ThreadPoolExecutor(max_workers=NODES) as run:
for name,n in self.node.items(): # {name: fablib_node)}
print(f’+++ setup_node: {name}’)
#self.setup_node(name,n,join) # blocking version
run.submit(self.setup_node,name,n,join)
run.shutdown()

(Yes I know I could do this by either running multiprocessing at a higher level or simply spawning jobs on the node and wait for completion and guessing I will need to do it this way)

[Paul Ruth]

That is a great question and we have been working on this.

The short answer is that the library as a whole is probably not thread safe.   However, the”node.execute()” method is really just a wrapper around a paramiko call and can be thread safe.   Also most of the “get” methods should be thread safe.

We have very rough (but working) version of a threaded execute call in a coming branch of fablib.   It’s not in its final form and interface will probably be simplified so I wouldn’t build too much based on this branch.  However, you could look at what we have and implement a small wrapper in you notebook that will accomplish your goal.

You can look here to see how we got the initial version to work: https://github.com/fabric-testbed/fabrictestbed-extensions/blob/fablib1.1.4dev1/fabrictestbed_extensions/fablib/node.py#:~:text=def-,execute_thread,-(self%2C 

One of the changes I want to make is to have a pool of theads.  Here is an example of fablib using a thread pool to get the interface names of from the VMs:  https://github.com/fabric-testbed/fabrictestbed-extensions/blob/1.2rc12+-pruth-working/fabrictestbed_extensions/fablib/slice.py#:~:text=def-,list_interfaces,-(self)%3A

Use these only as examples because they are not really ready for users, but they both work and I expect you can pull some ideas from them.

Paul

[Timothy Middelkoop]

Great – I’ll port my change back to that when it’s released (it ended up being not so hard to do it on the node). (I did and do not expect a fully thread-safe version of the library – I know how hard that can be).

[Author]

Posts