[Komal Thareja]

Hi Yuanjun,

We had leaked config on the switch which has been cleared by help from Network Team. Could you please try your slice again? Please let us know if you still see the issue.

Thanks,
Komal

[Komal Thareja]

User has confirmed in another post that this issue has been resolved.

SSH Connection Error: ChannelException(2, ‘Connect failed’)

[Komal Thareja]

User has confirmed that this is no longer an issue.


HI, Komal

The problem has been solved. I believe we do not need the meeting today.

Thank you so much for your help!

Best Regards,
Yuanjun Dai


[Komal Thareja]

Subject: API Behavior Verification

Hi Yuanjun,

I have verified this, and the API is functioning as intended.

fablib.list_sites() retrieves resource information from the testbed. Once fetched, users can display data for a specific site using fablib.show_site(). Please note that fablib.show_site() only presents site information that was previously retrieved via fablib.list_sites().

To refresh the resource information, you need to call fablib.list_sites(update=True), followed by fablib.show_site().

Please check this in your code, and let us know if you continue to experience any issues.

Thanks,
Komal

[Komal Thareja]

Hi Yuanjun,

Both the issues with upload_file and this point to SSH access to the VMs. It could be because of the expired bastion keys or a configuration issue.
Is it possible for a quick zoom meeting to resolve this? Please let me know if that works.

Thanks,
Komal

[Komal Thareja]

Hi Ilya,

I checked the host and see a similar output. Will check with Mert regarding the reboot.


0000:25:00.0 Network controller: Xilinx Corporation Device 903f
Subsystem: Xilinx Corporation Device 0007
Physical Slot: 2-1
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-

Thanks,
Komal

[Komal Thareja]

Hi Ilya,

Hussam helped recover the WASH worker. Could you please try your slice again on WASH?

Thanks,

Komal

[Komal Thareja]

Could you please check for any error messages in /tmp/fablib/fablib.log file when upload_file() is invoked? If possible, share this file?

This file should exist on the system from where the fablib is being invoked.

Thanks,

Komal

[Komal Thareja]

Thank you for sharing your observation. I will investigate this more and share upate.

Thanks,

Komal

[Komal Thareja]

Could you run fablib.show_config() to display the log file in use?

Additionally, you can specify the log file location in the fabric_rc file, which may assist in debugging. Upload failures often occur if the bastion keys have expired, so please verify that your bastion keys are still valid.

Thanks,

Komal

[Komal Thareja]

Hi Yuanjun,

This behavior is intentional; resource information is retrieved from cached data. Due to the scale of the system, updating it after each create or delete operation would be resource-intensive. Instead, the cache is refreshed at regular intervals, currently every 30 minutes.

Thanks,

Komal

[Komal Thareja]

/tmp/fablib/fablib.log log file exists on JH container. Could you please check and share that?

Thanks,

Komal

[Komal Thareja]

Hi Yuanjun,

Could you please try using upload_directory API and let us know if that works?

Also, are you able to SSH to the VMs? Could you please check if you see any errors in /tmp/fablib/fablib.log ?

If so, please share the log file.

Thanks,

Komal

[Komal Thareja]

Hi Ilya,

VM timedout on WASH for this slice. Hypervisor on the worker is currently down. Working to recover it. Will keep you posted.


+----+----------------------------+-----------------+-----------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+----+----------------------------+-----------------+-----------+-------+
| 1 | wash-w2.fabric-testbed.net | QEMU | 10.1.1.12 | down |
| 2 | wash-w3.fabric-testbed.net | QEMU | 10.1.1.13 | up |
| 3 | wash-w1.fabric-testbed.net | QEMU | 10.1.1.11 | up |
+----+----------------------------+-----------------+-----------+-------+

Thanks,

Komal

[Komal Thareja]

Hey Yuanjun,

You don’t need to have slice_key and slice_key.pub files. By default, Fablib expects an fabric_rc environment file to be present in the $HOME/work/fabric_config directory. A typical fabric_rc file contains the following:


export FABRIC_ORCHESTRATOR_HOST=orchestrator.fabric-testbed.net
export FABRIC_CREDMGR_HOST=cm.fabric-testbed.net
export FABRIC_CORE_API_HOST=uis.fabric-testbed.net
export FABRIC_AM_HOST=artifacts.fabric-testbed.net
export FABRIC_TOKEN_LOCATION=/Users/kthare10/work/id_token_maint.json
export FABRIC_PROJECT_ID=
export FABRIC_BASTION_HOST=bastion-renc-1.fabric-testbed.net
export FABRIC_BASTION_USERNAME=
export FABRIC_BASTION_KEY_LOCATION=/Users/kthare10/.ssh/bastion-prod-2
export FABRIC_SLICE_PUBLIC_KEY_FILE=/Users/kthare10/.ssh/id_rsa.pub
export FABRIC_SLICE_PRIVATE_KEY_FILE=/Users/kthare10/.ssh/id_rsa
export FABRIC_AVOID=
export FABRIC_SSH_COMMAND_LINE=ssh -i {{ _self_.private_ssh_key_file }} -F /Users/kthare10/work/fabric_config/ssh_config {{ _self_.username }}@{{ _self_.management_ip }}
export FABRIC_LOG_LEVEL=INFO
export FABRIC_LOG_FILE=/tmp/fablib/fablib.log
export FABRIC_BASTION_SSH_CONFIG_FILE=/Users/kthare10/work/fabric_config/ssh_config


It looks like you’re running the Fablib API directly from your laptop. I recommend reviewing the documentation for guidance on the two available approaches. You can find the relevant information here:

• Install Fablib API on your laptop
• Launch Docker Container

Thanks,

Komal

[Author]

Posts