[Komal Thareja]

User has confirmed that this is no longer an issue.


HI, Komal

The problem has been solved. I believe we do not need the meeting today.

Thank you so much for your help!

Best Regards,
Yuanjun Dai


[Komal Thareja]

Subject: API Behavior Verification

Hi Yuanjun,

I have verified this, and the API is functioning as intended.

fablib.list_sites() retrieves resource information from the testbed. Once fetched, users can display data for a specific site using fablib.show_site(). Please note that fablib.show_site() only presents site information that was previously retrieved via fablib.list_sites().

To refresh the resource information, you need to call fablib.list_sites(update=True), followed by fablib.show_site().

Please check this in your code, and let us know if you continue to experience any issues.

Thanks,
Komal

[Komal Thareja]

Hi Yuanjun,

Both the issues with upload_file and this point to SSH access to the VMs. It could be because of the expired bastion keys or a configuration issue.
Is it possible for a quick zoom meeting to resolve this? Please let me know if that works.

Thanks,
Komal

[Komal Thareja]

Hi Ilya,

I checked the host and see a similar output. Will check with Mert regarding the reboot.


0000:25:00.0 Network controller: Xilinx Corporation Device 903f
Subsystem: Xilinx Corporation Device 0007
Physical Slot: 2-1
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-

Thanks,
Komal

[Komal Thareja]

Hi Ilya,

Hussam helped recover the WASH worker. Could you please try your slice again on WASH?

Thanks,

Komal

[Komal Thareja]

Could you please check for any error messages in /tmp/fablib/fablib.log file when upload_file() is invoked? If possible, share this file?

This file should exist on the system from where the fablib is being invoked.

Thanks,

Komal

[Komal Thareja]

Thank you for sharing your observation. I will investigate this more and share upate.

Thanks,

Komal

[Komal Thareja]

Could you run fablib.show_config() to display the log file in use?

Additionally, you can specify the log file location in the fabric_rc file, which may assist in debugging. Upload failures often occur if the bastion keys have expired, so please verify that your bastion keys are still valid.

Thanks,

Komal

[Komal Thareja]

Hi Yuanjun,

This behavior is intentional; resource information is retrieved from cached data. Due to the scale of the system, updating it after each create or delete operation would be resource-intensive. Instead, the cache is refreshed at regular intervals, currently every 30 minutes.

Thanks,

Komal

[Komal Thareja]

/tmp/fablib/fablib.log log file exists on JH container. Could you please check and share that?

Thanks,

Komal

[Komal Thareja]

Hi Yuanjun,

Could you please try using upload_directory API and let us know if that works?

Also, are you able to SSH to the VMs? Could you please check if you see any errors in /tmp/fablib/fablib.log ?

If so, please share the log file.

Thanks,

Komal

[Komal Thareja]

Hi Ilya,

VM timedout on WASH for this slice. Hypervisor on the worker is currently down. Working to recover it. Will keep you posted.


+----+----------------------------+-----------------+-----------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+----+----------------------------+-----------------+-----------+-------+
| 1 | wash-w2.fabric-testbed.net | QEMU | 10.1.1.12 | down |
| 2 | wash-w3.fabric-testbed.net | QEMU | 10.1.1.13 | up |
| 3 | wash-w1.fabric-testbed.net | QEMU | 10.1.1.11 | up |
+----+----------------------------+-----------------+-----------+-------+

Thanks,

Komal

[Komal Thareja]

Hey Yuanjun,

You don’t need to have slice_key and slice_key.pub files. By default, Fablib expects an fabric_rc environment file to be present in the $HOME/work/fabric_config directory. A typical fabric_rc file contains the following:


export FABRIC_ORCHESTRATOR_HOST=orchestrator.fabric-testbed.net
export FABRIC_CREDMGR_HOST=cm.fabric-testbed.net
export FABRIC_CORE_API_HOST=uis.fabric-testbed.net
export FABRIC_AM_HOST=artifacts.fabric-testbed.net
export FABRIC_TOKEN_LOCATION=/Users/kthare10/work/id_token_maint.json
export FABRIC_PROJECT_ID=
export FABRIC_BASTION_HOST=bastion-renc-1.fabric-testbed.net
export FABRIC_BASTION_USERNAME=
export FABRIC_BASTION_KEY_LOCATION=/Users/kthare10/.ssh/bastion-prod-2
export FABRIC_SLICE_PUBLIC_KEY_FILE=/Users/kthare10/.ssh/id_rsa.pub
export FABRIC_SLICE_PRIVATE_KEY_FILE=/Users/kthare10/.ssh/id_rsa
export FABRIC_AVOID=
export FABRIC_SSH_COMMAND_LINE=ssh -i {{ _self_.private_ssh_key_file }} -F /Users/kthare10/work/fabric_config/ssh_config {{ _self_.username }}@{{ _self_.management_ip }}
export FABRIC_LOG_LEVEL=INFO
export FABRIC_LOG_FILE=/tmp/fablib/fablib.log
export FABRIC_BASTION_SSH_CONFIG_FILE=/Users/kthare10/work/fabric_config/ssh_config


It looks like you’re running the Fablib API directly from your laptop. I recommend reviewing the documentation for guidance on the two available approaches. You can find the relevant information here:

• Install Fablib API on your laptop
• Launch Docker Container

Thanks,

Komal

[Komal Thareja]

Hi Yuanjun,

Could you please share your fabric_rc file for your environment?

Thanks,

Komal

[Komal Thareja]

Hi Sepideh,

Is this still an issue? If so, could you please share your slice ID?

Also, this may happen because of expired bastion keys. Could you please run the notebook:  jupyter_examples-rel1.7.1/configure_and_validate.ipynb ? This notebook will renew your keys if the keys are expired.

Please let us know if this helped.

Thanks,

Komal

[Author]

Posts