[Komal Thareja]

Hi Yuanjun,

Both the issues with upload_file and this point to SSH access to the VMs. It could be because of the expired bastion keys or a configuration issue.
Is it possible for a quick zoom meeting to resolve this? Please let me know if that works.

Thanks,
Komal

[Komal Thareja]

Hi Ilya,

I checked the host and see a similar output. Will check with Mert regarding the reboot.


0000:25:00.0 Network controller: Xilinx Corporation Device 903f
Subsystem: Xilinx Corporation Device 0007
Physical Slot: 2-1
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-

Thanks,
Komal

[Komal Thareja]

Hi Ilya,

Hussam helped recover the WASH worker. Could you please try your slice again on WASH?

Thanks,

Komal

[Komal Thareja]

Could you please check for any error messages in /tmp/fablib/fablib.log file when upload_file() is invoked? If possible, share this file?

This file should exist on the system from where the fablib is being invoked.

Thanks,

Komal

[Komal Thareja]

Thank you for sharing your observation. I will investigate this more and share upate.

Thanks,

Komal

[Komal Thareja]

Could you run fablib.show_config() to display the log file in use?

Additionally, you can specify the log file location in the fabric_rc file, which may assist in debugging. Upload failures often occur if the bastion keys have expired, so please verify that your bastion keys are still valid.

Thanks,

Komal

[Komal Thareja]

Hi Yuanjun,

This behavior is intentional; resource information is retrieved from cached data. Due to the scale of the system, updating it after each create or delete operation would be resource-intensive. Instead, the cache is refreshed at regular intervals, currently every 30 minutes.

Thanks,

Komal

[Komal Thareja]

/tmp/fablib/fablib.log log file exists on JH container. Could you please check and share that?

Thanks,

Komal

[Komal Thareja]

Hi Yuanjun,

Could you please try using upload_directory API and let us know if that works?

Also, are you able to SSH to the VMs? Could you please check if you see any errors in /tmp/fablib/fablib.log ?

If so, please share the log file.

Thanks,

Komal

[Komal Thareja]

Hi Ilya,

VM timedout on WASH for this slice. Hypervisor on the worker is currently down. Working to recover it. Will keep you posted.


+----+----------------------------+-----------------+-----------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+----+----------------------------+-----------------+-----------+-------+
| 1 | wash-w2.fabric-testbed.net | QEMU | 10.1.1.12 | down |
| 2 | wash-w3.fabric-testbed.net | QEMU | 10.1.1.13 | up |
| 3 | wash-w1.fabric-testbed.net | QEMU | 10.1.1.11 | up |
+----+----------------------------+-----------------+-----------+-------+

Thanks,

Komal

[Komal Thareja]

Hey Yuanjun,

You don’t need to have slice_key and slice_key.pub files. By default, Fablib expects an fabric_rc environment file to be present in the $HOME/work/fabric_config directory. A typical fabric_rc file contains the following:


export FABRIC_ORCHESTRATOR_HOST=orchestrator.fabric-testbed.net
export FABRIC_CREDMGR_HOST=cm.fabric-testbed.net
export FABRIC_CORE_API_HOST=uis.fabric-testbed.net
export FABRIC_AM_HOST=artifacts.fabric-testbed.net
export FABRIC_TOKEN_LOCATION=/Users/kthare10/work/id_token_maint.json
export FABRIC_PROJECT_ID=
export FABRIC_BASTION_HOST=bastion-renc-1.fabric-testbed.net
export FABRIC_BASTION_USERNAME=
export FABRIC_BASTION_KEY_LOCATION=/Users/kthare10/.ssh/bastion-prod-2
export FABRIC_SLICE_PUBLIC_KEY_FILE=/Users/kthare10/.ssh/id_rsa.pub
export FABRIC_SLICE_PRIVATE_KEY_FILE=/Users/kthare10/.ssh/id_rsa
export FABRIC_AVOID=
export FABRIC_SSH_COMMAND_LINE=ssh -i {{ _self_.private_ssh_key_file }} -F /Users/kthare10/work/fabric_config/ssh_config {{ _self_.username }}@{{ _self_.management_ip }}
export FABRIC_LOG_LEVEL=INFO
export FABRIC_LOG_FILE=/tmp/fablib/fablib.log
export FABRIC_BASTION_SSH_CONFIG_FILE=/Users/kthare10/work/fabric_config/ssh_config


It looks like you’re running the Fablib API directly from your laptop. I recommend reviewing the documentation for guidance on the two available approaches. You can find the relevant information here:

• Install Fablib API on your laptop
• Launch Docker Container

Thanks,

Komal

[Komal Thareja]

Hi Yuanjun,

Could you please share your fabric_rc file for your environment?

Thanks,

Komal

[Komal Thareja]

Hi Sepideh,

Is this still an issue? If so, could you please share your slice ID?

Also, this may happen because of expired bastion keys. Could you please run the notebook:  jupyter_examples-rel1.7.1/configure_and_validate.ipynb ? This notebook will renew your keys if the keys are expired.

Please let us know if this helped.

Thanks,

Komal

[Komal Thareja]

Hi Pinxiang,

GUI does not support automatic configuration of IP addresses or complex topologies. When creating a slice from GUI, user is expected to configure the IP addresses manually after logging into the VM. You are right in this case the interface would be enp7s0. This can also be confirmed by matching the MAC address shown in the GUI and the interface.

I would strongly encourage you to try JupyterHub, we have several example available there which might be very helpful.

Please follow the instructions here to setup your Jupyter Hub environment and create a simple slice.

Also, sharing instruction for creating K8s cluster on FABRIC (Example created by Professor Fraida Fund).

Please let us know if you run into any issues or have questions.

Thanks,

Komal

[Komal Thareja]

Also, please note FabNetv4 network service is like an internet within Fabric and does not provide external connectivity. Please check out more details about the Network Services offered by Fabric here.

FabNetv*Ext services do offer external connectivity but require special permission to be enabled which can be requested by the Project Lead.

Thanks,

Komal

[Author]

Posts