Hi, I'm using Zuul 4.7.0 and Nodepool 4.2.2. My nodes are all static. Now and then all nodes enter an infinite 'building' state. E.g. here is how building node 0000000113 starts (node_request: 300-0000000094):
2021-08-19 15:28:42,203 DEBUG nodepool.PoolWorker.static-vms-main: [e: 5a36f7ab18e54a11ad728fecab385be6] [node_request: 300-0000000094] Locking request 2021-08-19 15:28:42,219 INFO nodepool.PoolWorker.static-vms-main: [e: 5a36f7ab18e54a11ad728fecab385be6] [node_request: 300-0000000094] Assigning node request <NodeRequest {'state': 'requested', 'state_time': 1629386913.1936047, 'declined_by': [], 'node_types': ['centos7'], 'nodes': [], 'reuse': True, 'requestor': '79d78decd29a', 'requestor_data': {'build_set_uuid': '344e4dec0639481b85d62c0ab6efa0e9', 'tenant_name': 'nto7', 'pipeline_name': 'check', 'job_name': 'buildroot-build'}, 'provider': None, 'relative_priority': 0, 'event_id': '5a36f7ab18e54a11ad728fecab385be6', 'id': '300-0000000094', 'stat': ZnodeStat(czxid=209282, mzxid=209282, ctime=1629386913193, mtime=1629386913193, version=0, cversion=0, aversion=0, ephemeralOwner=72217169448337409, dataLength=400, numChildren=0, pzxid=209282)}> 2021-08-19 15:28:42,219 DEBUG nodepool.driver.NodeRequestHandler[9c4813b37c0b-PoolWorker.static-vms-main-aa0248e60f7d49c9b5065855fe7e0842]: [e: 5a36f7ab18e54a11ad728fecab385be6] [node_request: 300-0000000094] Accepting node request 2021-08-19 15:28:42,253 DEBUG nodepool.driver.NodeRequestHandler[9c4813b37c0b-PoolWorker.static-vms-main-aa0248e60f7d49c9b5065855fe7e0842]: [e: 5a36f7ab18e54a11ad728fecab385be6] [node_request: 300-0000000094] Locked building node 0000000113 for request 2021-08-19 15:28:42,276 DEBUG nodepool.driver.NodeRequestHandler[9c4813b37c0b-PoolWorker.static-vms-main-aa0248e60f7d49c9b5065855fe7e0842]: [e: 5a36f7ab18e54a11ad728fecab385be6] [node_request: 300-0000000094] Waiting for node 0000000113 to be ready 2021-08-19 15:28:42,277 DEBUG nodepool.PoolWorker.static-vms-main: Active requests: ['300-0000000078', '300-0000000079', '300-0000000080', '300-0000000081', '300-0000000082', '300-0000000083', '300-0000000084', '300-0000000085', '300-0000000086', '300-0000000087', '300-0000000088', '300-0000000089', '300-0000000090', '300-0000000091', '300-0000000092', '300-0000000093', '300-0000000094']
After two days node_request: 300-0000000094 still hanging in active requests (node 0000000113 is also seen in web UI in building state). To clean nodepool I have to restart nodepool once a week.
Actually this issue has been tracing me for more than a year (so it's not something that appeared in the latest nodepool versions). I hoped recent commit [1] (included in nodepool 4.2.2) might be relevant to fixing my issue but it has not changed anything.
[1] https://review.opendev.org/c/zuul/nodepool/+/790674