TACC always failing with insufficient resources:Disk#

[Nishanth Shyamkumar]

I am trying to obtain VMs on TACC site with minimal resources; 2 cores, 8GB RAM and 10GB hard disk. However, the slice submission always fails with:

Slice Exception: Slice Name: Traffic Listening Demo TACC, Slice ID: 30f9fb42-37be-420f-899e-082a41bfb735: Slice Exception: Slice Name: Traffic Listening Demo TACC, Slice ID: 30f9fb42-37be-420f-899e-082a41bfb735: Node: TACC_node4, Site: TACC, State: Closed, Insufficient resources : [disk]#

Even if I submit the slice with a 1GB hard disk request, it fails with the above error. Can someone explain what this error means and how to fix it ?

[Komal Thareja]

Hi Nishanth,

The error Insufficient resources : [disk] implies that there is not enough disk available on the host on which your VM is being requested. Looking at your slice, following VM requesting a ConnectX5 is being rejected as it maps to tacc-w4  There is not enough disk available on tacc-w4 to accomodate your VM hence the failure.


Reservation ID: 478b2a91-5a02-4cf0-9bcd-de04c3b873ea Slice ID: 30f9fb42-37be-420f-899e-082a41bfb735
Resource Type: VM Notices: Reservation 478b2a91-5a02-4cf0-9bcd-de04c3b873ea (Slice Traffic Listening Demo TACC(30f9fb42-37be-420f-899e-082a41bfb735) Graph Id:a58b7bc7-55d6-42e9-b457-5a8a32ebebc9 Owner:nshyamkumar@iit.edu) is in state (Closed,None_) (Last ticket update: Insufficient resources : ['disk'])
Start: 2024-06-05 17:55:24 +0000 End: 2024-06-06 17:55:23 +0000 Requested End: 2024-06-06 17:55:23 +0000
Units: 1 State: Closed Pending State: None_
Predecessors
Sliver: {'node_id': '9a579143-79b2-44fb-bacb-e6a5db4da3bf', 'capacities': '{ core: 2 , ram: 8 G, disk: 1 G}', 'capacity_hints': '{ instance_type: fabric.c2.m8.d10}', 'image_ref': 'default_ubuntu_20', 'image_type': 'qcow2', 'name': 'TACC_node4', 'reservation_info': '{"reservation_id": "478b2a91-5a02-4cf0-9bcd-de04c3b873ea", "reservation_state": "Closed"}', 'site': 'TACC', 'type': 'VM', 'user_data': '{"fablib_data": {"instantiated": "False", "run_update_commands": "False", "post_boot_commands": [], "post_update_commands": []}}'}
Component: {'node_id': '670d117f-19ac-477b-bff7-36ac4e90107a', 'details': 'Mellanox ConnectX-5 Dual Port 10/25GbE', 'model': 'ConnectX-5', 'name': 'TACC_node4-pmnic_2', 'type': 'SmartNIC', 'user_data': '{}'}
NS: {'node_id': 'adeede90-a808-45a6-8e1e-8c8de7a4ee6e', 'layer': 'L2', 'name': 'TACC_node4-TACC_node4-pmnic_2-l2ovs', 'site': 'TACC', 'type': 'OVS'}
IFS: {'node_id': '2f9a52b4-3108-48f2-b0f9-e0ccd7716cdc', 'capacities': '{ bw: 25 Gbps, unit: 1 }', 'labels': '{ local_name: p1}', 'name': 'TACC_node4-pmnic_2-p1', 'type': 'DedicatedPort', 'user_data': '{"fablib_data": {"mode": "config"}}'}
IFS: {'node_id': 'b6c42c3e-a570-4ed1-b633-607e90777f34', 'capacities': '{ bw: 25 Gbps, unit: 1 }', 'labels': '{ local_name: p2}', 'name': 'TACC_node4-pmnic_2-p2', 'type': 'DedicatedPort', 'user_data': '{"fablib_data": {"mode": "config"}}'}


Thanks,

Komal

[Nishanth Shyamkumar]

Thanks, so it does indeed stand for disk space.

When I look at the graphical stats on the Fabric Portal, it mentions that TACC has 103263/107463 GB free (it may not be the latest info, but I don’t think it varies by much). How can I ask Fabric to assign my VM on an underlying server where there is enough hard disk space ?

[Komal Thareja]

Portal currently display overall site TACC disk usage which is combined disk space on all the hosts at TACC. Control Framework determines possible candidate nodes for your VM on the basis of the resources requested.

Your slice is requesting for a VM with CX5, which in case of TACC are only available on tacc-w4 and hence it is trying to allocate it on tacc-w4 but fails due to no disk space. I would recommend using a different site than TACC.

Also, we are working on improving the Resource Usage display to show per Host level information.

Thanks,

Komal

 

[Author]

Posts