Timeout while creating slice

[Nishanth Shyamkumar]

I am creating a slice that encompasses 8 sites, and when I submit the slice request it fails with a timeout exceeded exception. I am using the following command:

pmslice.submit(progress=False, wait_timeout=2400, wait_interval=120)

However, internally this method is calling self.wait() which is set to a default timeout of 360 seconds. The wait_timeout parameter in the submit method doesn’t seem to have the desired effect of increasing the timeout period. Is there a way that I can increase the timeout period for the wait() method?

The error log:

Timeout exceeded (360 sec). Slice: Traffic Listening Slice (Configuring)

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
Cell In[6], line 111
    109 try:
    110     #print("dummy stub")
--> 111     pmslice.submit(progress=False, wait_timeout=2400, wait_interval=120)
    112 except Exception as e:

File /opt/conda/lib/python3.10/site-packages/fabrictestbed_extensions/fablib/slice.py:2040, in Slice.submit(self, wait, wait_timeout, wait_interval, progress, wait_jupyter, post_boot_config, wait_ssh, extra_ssh_keys)
   2038 self.update()
-> 2040 self.wait()
   2042 if wait_ssh:

File /opt/conda/lib/python3.10/site-packages/fabrictestbed_extensions/fablib/slice.py:1518, in Slice.wait(self, timeout, interval, progress)
   1517 if time.time() >= timeout_start + timeout:
-> 1518     raise Exception(
   1519         " Timeout exceeded ({} sec). Slice: {} ({})".format(
   1520             timeout, slice.name, slice.state
   1521         )
   1522     )
   1524 # Update the fim topology (wait to avoid get topology bug)
   1525 # time.sleep(interval)

Exception:  Timeout exceeded (360 sec). Slice: Traffic Listening Slice (Configuring)

[Paul Ruth]

My first thought is that something in the slice is going to fail and increasing the timeout is not going to help.  I could be wrong but lets try a test with several of the submit tasks separated.

Try something like the following. If you make each of these steps in a separate notebook cell you can re-run the wait calls as many times as necessary.  The slice.wait(progress=True) call may time out for you. Just re-run it as many times as you need.

slice.submit(progress=False)
slice.wait(progress=True)
slice.wait_ssh(progress=True)
slice.post_boot_config()

Let me know if this works.

Paul

[Nishanth Shyamkumar]

Thanks Paul, I looked into the source code for this and I saw that the ‘main’ branch actually includes a change that propagates the wait_timeout parameter to the self.wait() function call. However it’s available only on the ‘beyond bleeding’ while I was testing on the ‘bleeding’ framework.

I am still sticking with the ‘bleeding’ framework as of now because the code is structured in such a way that if I set progress=True (which is the default), then the wait_timeout propagates to self.wait(). I was testing earlier with progress=False, since I didn’t want the overhead of GUI representation of the data, but I got to use it for now at least.

I tested with progress=True and wait_timeout=2400 and it works for now. The slice submission takes between 1000 to 1500 seconds to complete, but it does succeed in the end.

[Author]

Posts