- This topic has 2 replies, 2 voices, and was last updated 11 months, 1 week ago by
.
-
Posts
-
I am creating a slice that encompasses 8 sites, and when I submit the slice request it fails with a timeout exceeded exception. I am using the following command:
pmslice.submit(progress=False, wait_timeout=2400, wait_interval=120)
However, internally this method is calling self.wait() which is set to a default timeout of 360 seconds. The wait_timeout parameter in the submit method doesn’t seem to have the desired effect of increasing the timeout period. Is there a way that I can increase the timeout period for the wait() method?
The error log:
Timeout exceeded (360 sec). Slice: Traffic Listening Slice (Configuring)
--------------------------------------------------------------------------- Exception Traceback (most recent call last) Cell In[6], line 111 109 try: 110 #print("dummy stub") --> 111 pmslice.submit(progress=False, wait_timeout=2400, wait_interval=120) 112 except Exception as e: File /opt/conda/lib/python3.10/site-packages/fabrictestbed_extensions/fablib/slice.py:2040, in Slice.submit(self, wait, wait_timeout, wait_interval, progress, wait_jupyter, post_boot_config, wait_ssh, extra_ssh_keys) 2038 self.update() -> 2040 self.wait() 2042 if wait_ssh: File /opt/conda/lib/python3.10/site-packages/fabrictestbed_extensions/fablib/slice.py:1518, in Slice.wait(self, timeout, interval, progress) 1517 if time.time() >= timeout_start + timeout: -> 1518 raise Exception( 1519 " Timeout exceeded ({} sec). Slice: {} ({})".format( 1520 timeout, slice.name, slice.state 1521 ) 1522 ) 1524 # Update the fim topology (wait to avoid get topology bug) 1525 # time.sleep(interval) Exception: Timeout exceeded (360 sec). Slice: Traffic Listening Slice (Configuring)
My first thought is that something in the slice is going to fail and increasing the timeout is not going to help. I could be wrong but lets try a test with several of the submit tasks separated.
Try something like the following. If you make each of these steps in a separate notebook cell you can re-run the wait calls as many times as necessary. The
slice.wait(progress=True)call may time out for you. Just re-run it as many times as you need.slice.submit(progress=False) slice.wait(progress=True) slice.wait_ssh(progress=True) slice.post_boot_config()Let me know if this works.
Paul
Thanks Paul, I looked into the source code for this and I saw that the ‘main’ branch actually includes a change that propagates the wait_timeout parameter to the self.wait() function call. However it’s available only on the ‘beyond bleeding’ while I was testing on the ‘bleeding’ framework.
I am still sticking with the ‘bleeding’ framework as of now because the code is structured in such a way that if I set progress=True (which is the default), then the wait_timeout propagates to self.wait(). I was testing earlier with progress=False, since I didn’t want the overhead of GUI representation of the data, but I got to use it for now at least.
I tested with progress=True and wait_timeout=2400 and it works for now. The slice submission takes between 1000 to 1500 seconds to complete, but it does succeed in the end.
- You must be logged in to reply to this topic.