1. Komal Thareja

Komal Thareja

Forum Replies Created

Viewing 15 posts - 136 through 150 (of 487 total)
  • Author
    Posts
  • Komal Thareja
    Participant

      Hey Yuanjun,

      You don’t need to have slice_key and slice_key.pub files. By default, Fablib expects an fabric_rc environment file to be present in the $HOME/work/fabric_config directory. A typical fabric_rc file contains the following:


      export FABRIC_ORCHESTRATOR_HOST=orchestrator.fabric-testbed.net
      export FABRIC_CREDMGR_HOST=cm.fabric-testbed.net
      export FABRIC_CORE_API_HOST=uis.fabric-testbed.net
      export FABRIC_AM_HOST=artifacts.fabric-testbed.net
      export FABRIC_TOKEN_LOCATION=/Users/kthare10/work/id_token_maint.json
      export FABRIC_PROJECT_ID=
      export FABRIC_BASTION_HOST=bastion-renc-1.fabric-testbed.net
      export FABRIC_BASTION_USERNAME=
      export FABRIC_BASTION_KEY_LOCATION=/Users/kthare10/.ssh/bastion-prod-2
      export FABRIC_SLICE_PUBLIC_KEY_FILE=/Users/kthare10/.ssh/id_rsa.pub
      export FABRIC_SLICE_PRIVATE_KEY_FILE=/Users/kthare10/.ssh/id_rsa
      export FABRIC_AVOID=
      export FABRIC_SSH_COMMAND_LINE=ssh -i {{ _self_.private_ssh_key_file }} -F /Users/kthare10/work/fabric_config/ssh_config {{ _self_.username }}@{{ _self_.management_ip }}
      export FABRIC_LOG_LEVEL=INFO
      export FABRIC_LOG_FILE=/tmp/fablib/fablib.log
      export FABRIC_BASTION_SSH_CONFIG_FILE=/Users/kthare10/work/fabric_config/ssh_config

      It looks like you’re running the Fablib API directly from your laptop. I recommend reviewing the documentation for guidance on the two available approaches. You can find the relevant information here:

      Thanks,

      Komal

      Komal Thareja
      Participant

        Hi Yuanjun,

        Could you please share your fabric_rc file for your environment?

        Thanks,

        Komal

        in reply to: SSH Login Issue with Client #7994
        Komal Thareja
        Participant

          Hi Sepideh,

          Is this still an issue? If so, could you please share your slice ID?

          Also, this may happen because of expired bastion keys. Could you please run the notebook:  jupyter_examples-rel1.7.1/configure_and_validate.ipynb ? This notebook will renew your keys if the keys are expired.

          Please let us know if this helped.

          Thanks,

          Komal

          Komal Thareja
          Participant

            Hi Pinxiang,

            GUI does not support automatic configuration of IP addresses or complex topologies. When creating a slice from GUI, user is expected to configure the IP addresses manually after logging into the VM. You are right in this case the interface would be enp7s0. This can also be confirmed by matching the MAC address shown in the GUI and the interface.

            I would strongly encourage you to try JupyterHub, we have several example available there which might be very helpful.

            Please follow the instructions here to setup your Jupyter Hub environment and create a simple slice.

            Also, sharing instruction for creating K8s cluster on FABRIC (Example created by Professor Fraida Fund).

            Please let us know if you run into any issues or have questions.

            Thanks,

            Komal

            Komal Thareja
            Participant

              Also, please note FabNetv4 network service is like an internet within Fabric and does not provide external connectivity. Please check out more details about the Network Services offered by Fabric here.

              FabNetv*Ext services do offer external connectivity but require special permission to be enabled which can be requested by the Project Lead.

              Thanks,

              Komal

              Komal Thareja
              Participant

                Hi Pinxiang,

                Looking at your slice, you have 3 VMs connected to FabNetv4 service as you mentioned. But the IP addresses are not configured on the respective interfaces on the VMs, hence the traffic does not pass.

                Could you please try Fabnetv4 example accessible via start_here.ipynb?

                FABNet IPv4 (Layer 3): Connect to FABRIC’s IPv4 internet – it has 3 options auto, manual and full auto.

                In the auto, and full auto, API takes care of configuring the IP addresses and traffic should pass on the IPv4 address while in the manual configuration, user is explicitly required to configure the IP addresses.

                Please feel free to reach out in case of questions or concers.

                Snippet from your VMs:


                root@3bb1005a-6a0f-4b52-9c07-75d453b50813-node1:~# ifconfig -a
                enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
                inet 10.30.6.153 netmask 255.255.254.0 broadcast 10.30.7.255
                inet6 2001:400:a100:3020:f816:3eff:fe23:bd75 prefixlen 64 scopeid 0x0<global>
                inet6 fe80::f816:3eff:fe23:bd75 prefixlen 64 scopeid 0x20<link>
                ether fa:16:3e:23:bd:75 txqueuelen 1000 (Ethernet)
                RX packets 364541 bytes 306898628 (306.8 MB)
                RX errors 0 dropped 0 overruns 0 frame 0
                TX packets 34407 bytes 3474334 (3.4 MB)
                TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

                enp7s0: flags=4098<BROADCAST,MULTICAST> mtu 1500
                ether 02:50:a9:17:fc:d4 txqueuelen 1000 (Ethernet)
                RX packets 0 bytes 0 (0.0 B)
                RX errors 0 dropped 0 overruns 0 frame 0
                TX packets 0 bytes 0 (0.0 B)
                TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

                lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
                inet 127.0.0.1 netmask 255.0.0.0
                inet6 ::1 prefixlen 128 scopeid 0x10<host>
                loop txqueuelen 1000 (Local Loopback)
                RX packets 662 bytes 115088 (115.0 KB)
                RX errors 0 dropped 0 overruns 0 frame 0
                TX packets 662 bytes 115088 (115.0 KB)
                TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

                ubuntu@05441f94-5e35-4981-97d3-1ed1dac3381e-node3:~$ ifconfig -a
                enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
                inet 10.30.6.231 netmask 255.255.254.0 broadcast 10.30.7.255
                inet6 fe80::f816:3eff:fec8:e21a prefixlen 64 scopeid 0x20<link>
                inet6 2001:400:a100:3020:f816:3eff:fec8:e21a prefixlen 64 scopeid 0x0<global>
                ether fa:16:3e:c8:e2:1a txqueuelen 1000 (Ethernet)
                RX packets 368197 bytes 307203695 (307.2 MB)
                RX errors 0 dropped 0 overruns 0 frame 0
                TX packets 37436 bytes 3716755 (3.7 MB)
                TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

                enp7s0: flags=4098<BROADCAST,MULTICAST> mtu 1500
                ether 02:fe:2e:df:af:a7 txqueuelen 1000 (Ethernet)
                RX packets 0 bytes 0 (0.0 B)
                RX errors 0 dropped 0 overruns 0 frame 0
                TX packets 0 bytes 0 (0.0 B)
                TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

                lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
                inet 127.0.0.1 netmask 255.0.0.0
                inet6 ::1 prefixlen 128 scopeid 0x10<host>
                loop txqueuelen 1000 (Local Loopback)
                RX packets 556 bytes 95893 (95.8 KB)
                RX errors 0 dropped 0 overruns 0 frame 0
                TX packets 556 bytes 95893 (95.8 KB)
                TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

                ubuntu@9ac56841-a123-4efa-9322-af75d3731819-node2:~$ ifconfig -a
                enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
                inet 10.30.6.23 netmask 255.255.254.0 broadcast 10.30.7.255
                inet6 2001:400:a100:3020:f816:3eff:fe62:510a prefixlen 64 scopeid 0x0<global>
                inet6 fe80::f816:3eff:fe62:510a prefixlen 64 scopeid 0x20<link>
                ether fa:16:3e:62:51:0a txqueuelen 1000 (Ethernet)
                RX packets 379700 bytes 308068634 (308.0 MB)
                RX errors 0 dropped 0 overruns 0 frame 0
                TX packets 46612 bytes 4697307 (4.6 MB)
                TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

                enp7s0: flags=4098<BROADCAST,MULTICAST> mtu 1500
                ether 02:ef:84:b8:fd:09 txqueuelen 1000 (Ethernet)
                RX packets 0 bytes 0 (0.0 B)
                RX errors 0 dropped 0 overruns 0 frame 0
                TX packets 0 bytes 0 (0.0 B)
                TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

                lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
                inet 127.0.0.1 netmask 255.0.0.0
                inet6 ::1 prefixlen 128 scopeid 0x10<host>
                loop txqueuelen 1000 (Local Loopback)
                RX packets 570 bytes 97991 (97.9 KB)
                RX errors 0 dropped 0 overruns 0 frame 0
                TX packets 570 bytes 97991 (97.9 KB)
                TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

                Thanks,

                Komal

                • This reply was modified 9 months, 2 weeks ago by Komal Thareja.
                Komal Thareja
                Participant

                  Hi Pinxiang,

                  Could you please share your Slice ID?

                  Thanks,

                  Komal

                  in reply to: issues setting up bastion keys #7971
                  Komal Thareja
                  Participant

                    Hi Sean,

                    Could you please share the snapshot of the screen from the portal?

                    Experiments -> Manage SSH Keys -> Bastion

                    Please include the Bastion User name shown on this screen as well.

                    In addition, could you please try running the notebook jupyter-examples-rel1.7.0/configure_and_validate.ipynb from the JH. This notebook validates your configuration, creates bastion and sliver keys if not present or expired. Please try a Hello example as well after this to verify if your keys and configuration is  working.

                    Thanks,

                    Komal

                    in reply to: interrupt handling in VMs #7969
                    Komal Thareja
                    Participant

                      Hi Nirmala,

                      We currently do not have an example available for this. But I plan to work on one after the holidays and will share an update with you once I have a working version.

                      Thanks,

                      Komal

                      in reply to: Unable to run Jupyter Notebook from laptop #7968
                      Komal Thareja
                      Participant

                        Hi Prateek,

                        We recently published the steps to launch a local JH container from your desktop and laptop.

                        Please consider giving this a try. Also, regarding your existing setup, could you please check fabric_rc is pointing to the correct token location where you have uploaded the newly generated token.

                        Another thing to verify would be to generate the token via an incognito browser window to ensure any stale cookies.

                        Please let me know if this helps or you still face issues!

                        Thanks,

                        Komal

                        in reply to: Unable to create EDC slice #7952
                        Komal Thareja
                        Participant

                          Hi Ilya,

                          EDC and NCSA are connected to the same switch and share the same /17 subnet for FabNetv4 allocation. Between these two sites EDC(65) and NCSA(63), we currently have 128 active FabNetv4 services provisioned leaving no available subnet. The error message returned is not user friendly. I will fix the error message in the 1.8 update.

                          Thanks,

                          Komal

                          in reply to: Hardware Steering – Connectx6 #7950
                          Komal Thareja
                          Participant

                            Hi Tanay,

                            I wanted to check if this issue is still unresolved. I haven’t had a chance to look into it yet, but I plan to review the documentation and experiment with a few approaches. I’ll share any updates or findings here after the holidays.

                            Thanks,
                            Komal

                            in reply to: Unable to access VMs #7941
                            Komal Thareja
                            Participant

                              Hi Rodrigo,

                              It’s possible that your bastion keys have expired. Could you please check the expiration of the keys from the Portal via Experiments -> Manage SSH Keys?

                              Also, please try running the notebook jupyter-examples-rel1.7.1/configure_and_validate.ipynb ?

                              This notebook shall regenerate the bastion keys if the keys have expired. Please very SSH access after that.

                              Please let us know if you still see errors.

                              Thanks,

                              Komal

                              in reply to: Setting up Kubernetes cluster on FABRIC #7928
                              Komal Thareja
                              Participant

                                Thank you so much @Fraida! Could we please request you to consider uploading this to Fabric Artifacts to enable other Fabric users to leverage this?

                                Appreciate your help with this!

                                Artifact Manager: https://artifacts.fabric-testbed.net/artifacts/

                                in reply to: Insufficient resources error despite available resources #7924
                                Komal Thareja
                                Participant

                                  Hi Jestus,

                                  This error occurs when the host capable of provisioning the requested resource has run out of cores and RAM. While the resource view provides cumulative information for the entire site, checking resource availability at the host level offers more precise insights. This is available on the portal for each site resource view and also can be checked via API as shown by list_hosts in example here.

                                  It’s possible that the combination of requested components (such as NICs or GPUs) maps to a host without sufficient cores or RAM, leading to the error you’ve encountered.

                                  We have an example notebook  (Additional Options: Validate Slice) available that allows you to validate resource availability beforehand using the API, which can be helpful prior to submitting a slice. Additionally, we’re working on changes to the allocation policy to better distribute VMs across hosts. This will help ensure that CPUs, RAM and disk are not fully allocated on single host which has SmartNICs and GPUs, minimizing such errors. These updates are planned for deployment in the January Release and should improve resource allocation.

                                  Thanks,
                                  Komal

                                Viewing 15 posts - 136 through 150 (of 487 total)