[ 466.927377] env[62575]: INFO neutron.common.config [-] Logging enabled! [ 466.927867] env[62575]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 466.927867] env[62575]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=62575) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 466.928093] env[62575]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 466.928185] env[62575]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 466.929009] env[62575]: WARNING oslo_policy.policy [None req-430c476a-fb94-4742-a813-6669a13aed94 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 466.929227] env[62575]: WARNING oslo_policy.policy [None req-430c476a-fb94-4742-a813-6669a13aed94 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 466.931322] env[62575]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-430c476a-fb94-4742-a813-6669a13aed94 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 466.972743] env[62575]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-430c476a-fb94-4742-a813-6669a13aed94 None None] Session token - acquiring [ 466.973511] env[62575]: DEBUG networking_nsxv3.common.locking [None req-430c476a-fb94-4742-a813-6669a13aed94 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62575) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 466.973709] env[62575]: DEBUG oslo_concurrency.lockutils [None req-430c476a-fb94-4742-a813-6669a13aed94 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62575) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 466.973835] env[62575]: DEBUG oslo_concurrency.lockutils [None req-430c476a-fb94-4742-a813-6669a13aed94 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62575) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 466.974236] env[62575]: DEBUG oslo_concurrency.lockutils [None req-430c476a-fb94-4742-a813-6669a13aed94 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62575) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 466.984511] env[62575]: DEBUG oslo_concurrency.lockutils [None req-430c476a-fb94-4742-a813-6669a13aed94 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62575) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 466.984752] env[62575]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-430c476a-fb94-4742-a813-6669a13aed94 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 466.984928] env[62575]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-430c476a-fb94-4742-a813-6669a13aed94 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=62575) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 468.991455] env[62575]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-430c476a-fb94-4742-a813-6669a13aed94 None None] Session token - acquiring [ 468.991917] env[62575]: DEBUG networking_nsxv3.common.locking [None req-430c476a-fb94-4742-a813-6669a13aed94 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62575) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 468.992128] env[62575]: DEBUG oslo_concurrency.lockutils [None req-430c476a-fb94-4742-a813-6669a13aed94 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62575) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 468.992255] env[62575]: DEBUG oslo_concurrency.lockutils [None req-430c476a-fb94-4742-a813-6669a13aed94 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62575) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 468.992542] env[62575]: DEBUG oslo_concurrency.lockutils [None req-430c476a-fb94-4742-a813-6669a13aed94 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62575) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 469.002564] env[62575]: DEBUG oslo_concurrency.lockutils [None req-430c476a-fb94-4742-a813-6669a13aed94 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62575) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 469.002813] env[62575]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-430c476a-fb94-4742-a813-6669a13aed94 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 469.002957] env[62575]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-430c476a-fb94-4742-a813-6669a13aed94 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=62575) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 471.005955] env[62575]: CRITICAL neutron [None req-430c476a-fb94-4742-a813-6669a13aed94 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 471.005955] env[62575]: ERROR neutron Traceback (most recent call last): [ 471.005955] env[62575]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 471.005955] env[62575]: ERROR neutron sys.exit(main()) [ 471.005955] env[62575]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 471.005955] env[62575]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 471.005955] env[62575]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 471.005955] env[62575]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 471.005955] env[62575]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 471.005955] env[62575]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 471.005955] env[62575]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 471.005955] env[62575]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 471.005955] env[62575]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 471.005955] env[62575]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 471.005955] env[62575]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 471.005955] env[62575]: ERROR neutron response = self.get(path=path, params=params) [ 471.006790] env[62575]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 471.006790] env[62575]: ERROR neutron raise RuntimeError(msg, last_err) [ 471.006790] env[62575]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 471.006790] env[62575]: ERROR neutron [ 473.463474] env[62607]: INFO neutron.common.config [-] Logging enabled! [ 473.463941] env[62607]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 473.464092] env[62607]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=62607) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 473.464250] env[62607]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 473.464346] env[62607]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 473.465229] env[62607]: WARNING oslo_policy.policy [None req-681489f2-a9c7-4142-840b-16a96fbf50de None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 473.465471] env[62607]: WARNING oslo_policy.policy [None req-681489f2-a9c7-4142-840b-16a96fbf50de None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 473.466656] env[62607]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-681489f2-a9c7-4142-840b-16a96fbf50de None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 473.483264] env[62607]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-681489f2-a9c7-4142-840b-16a96fbf50de None None] Session token - acquiring [ 473.484065] env[62607]: DEBUG networking_nsxv3.common.locking [None req-681489f2-a9c7-4142-840b-16a96fbf50de None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62607) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 473.484274] env[62607]: DEBUG oslo_concurrency.lockutils [None req-681489f2-a9c7-4142-840b-16a96fbf50de None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62607) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 473.484400] env[62607]: DEBUG oslo_concurrency.lockutils [None req-681489f2-a9c7-4142-840b-16a96fbf50de None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62607) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 473.484818] env[62607]: DEBUG oslo_concurrency.lockutils [None req-681489f2-a9c7-4142-840b-16a96fbf50de None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62607) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 473.494164] env[62607]: DEBUG oslo_concurrency.lockutils [None req-681489f2-a9c7-4142-840b-16a96fbf50de None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62607) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 473.494444] env[62607]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-681489f2-a9c7-4142-840b-16a96fbf50de None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 473.494597] env[62607]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-681489f2-a9c7-4142-840b-16a96fbf50de None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=62607) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 475.500541] env[62607]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-681489f2-a9c7-4142-840b-16a96fbf50de None None] Session token - acquiring [ 475.500981] env[62607]: DEBUG networking_nsxv3.common.locking [None req-681489f2-a9c7-4142-840b-16a96fbf50de None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62607) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 475.501099] env[62607]: DEBUG oslo_concurrency.lockutils [None req-681489f2-a9c7-4142-840b-16a96fbf50de None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62607) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 475.501187] env[62607]: DEBUG oslo_concurrency.lockutils [None req-681489f2-a9c7-4142-840b-16a96fbf50de None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62607) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 475.501466] env[62607]: DEBUG oslo_concurrency.lockutils [None req-681489f2-a9c7-4142-840b-16a96fbf50de None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62607) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 475.510294] env[62607]: DEBUG oslo_concurrency.lockutils [None req-681489f2-a9c7-4142-840b-16a96fbf50de None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62607) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 475.510527] env[62607]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-681489f2-a9c7-4142-840b-16a96fbf50de None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 475.510666] env[62607]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-681489f2-a9c7-4142-840b-16a96fbf50de None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=62607) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 477.513593] env[62607]: CRITICAL neutron [None req-681489f2-a9c7-4142-840b-16a96fbf50de None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 477.513593] env[62607]: ERROR neutron Traceback (most recent call last): [ 477.513593] env[62607]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 477.513593] env[62607]: ERROR neutron sys.exit(main()) [ 477.513593] env[62607]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 477.513593] env[62607]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 477.513593] env[62607]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 477.513593] env[62607]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 477.513593] env[62607]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 477.513593] env[62607]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 477.513593] env[62607]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 477.513593] env[62607]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 477.513593] env[62607]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 477.513593] env[62607]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 477.513593] env[62607]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 477.513593] env[62607]: ERROR neutron response = self.get(path=path, params=params) [ 477.514300] env[62607]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 477.514300] env[62607]: ERROR neutron raise RuntimeError(msg, last_err) [ 477.514300] env[62607]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 477.514300] env[62607]: ERROR neutron [ 479.949977] env[62630]: INFO neutron.common.config [-] Logging enabled! [ 479.950379] env[62630]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 479.950474] env[62630]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=62630) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 479.950692] env[62630]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 479.950783] env[62630]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 479.951602] env[62630]: WARNING oslo_policy.policy [None req-5cee7da4-2d34-4797-96bb-cc0578011481 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 479.951831] env[62630]: WARNING oslo_policy.policy [None req-5cee7da4-2d34-4797-96bb-cc0578011481 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 479.953072] env[62630]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-5cee7da4-2d34-4797-96bb-cc0578011481 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 479.967107] env[62630]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5cee7da4-2d34-4797-96bb-cc0578011481 None None] Session token - acquiring [ 479.967882] env[62630]: DEBUG networking_nsxv3.common.locking [None req-5cee7da4-2d34-4797-96bb-cc0578011481 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62630) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 479.968095] env[62630]: DEBUG oslo_concurrency.lockutils [None req-5cee7da4-2d34-4797-96bb-cc0578011481 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62630) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 479.968225] env[62630]: DEBUG oslo_concurrency.lockutils [None req-5cee7da4-2d34-4797-96bb-cc0578011481 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62630) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 479.968639] env[62630]: DEBUG oslo_concurrency.lockutils [None req-5cee7da4-2d34-4797-96bb-cc0578011481 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62630) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 479.977605] env[62630]: DEBUG oslo_concurrency.lockutils [None req-5cee7da4-2d34-4797-96bb-cc0578011481 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62630) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 479.977855] env[62630]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5cee7da4-2d34-4797-96bb-cc0578011481 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 479.978009] env[62630]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5cee7da4-2d34-4797-96bb-cc0578011481 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=62630) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 481.983937] env[62630]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5cee7da4-2d34-4797-96bb-cc0578011481 None None] Session token - acquiring [ 481.984351] env[62630]: DEBUG networking_nsxv3.common.locking [None req-5cee7da4-2d34-4797-96bb-cc0578011481 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62630) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 481.984470] env[62630]: DEBUG oslo_concurrency.lockutils [None req-5cee7da4-2d34-4797-96bb-cc0578011481 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62630) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 481.984599] env[62630]: DEBUG oslo_concurrency.lockutils [None req-5cee7da4-2d34-4797-96bb-cc0578011481 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62630) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 481.984908] env[62630]: DEBUG oslo_concurrency.lockutils [None req-5cee7da4-2d34-4797-96bb-cc0578011481 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62630) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 481.993867] env[62630]: DEBUG oslo_concurrency.lockutils [None req-5cee7da4-2d34-4797-96bb-cc0578011481 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62630) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 481.994110] env[62630]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5cee7da4-2d34-4797-96bb-cc0578011481 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 481.994248] env[62630]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5cee7da4-2d34-4797-96bb-cc0578011481 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=62630) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 483.996928] env[62630]: CRITICAL neutron [None req-5cee7da4-2d34-4797-96bb-cc0578011481 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 483.996928] env[62630]: ERROR neutron Traceback (most recent call last): [ 483.996928] env[62630]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 483.996928] env[62630]: ERROR neutron sys.exit(main()) [ 483.996928] env[62630]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 483.996928] env[62630]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 483.996928] env[62630]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 483.996928] env[62630]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 483.996928] env[62630]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 483.996928] env[62630]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 483.996928] env[62630]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 483.996928] env[62630]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 483.996928] env[62630]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 483.996928] env[62630]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 483.996928] env[62630]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 483.996928] env[62630]: ERROR neutron response = self.get(path=path, params=params) [ 483.997722] env[62630]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 483.997722] env[62630]: ERROR neutron raise RuntimeError(msg, last_err) [ 483.997722] env[62630]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 483.997722] env[62630]: ERROR neutron [ 486.454664] env[62677]: INFO neutron.common.config [-] Logging enabled! [ 486.455249] env[62677]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 486.455374] env[62677]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=62677) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 486.455661] env[62677]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 486.455809] env[62677]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 486.456686] env[62677]: WARNING oslo_policy.policy [None req-2fb4c659-ed50-4bb1-984b-ddbb39f48c51 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 486.456966] env[62677]: WARNING oslo_policy.policy [None req-2fb4c659-ed50-4bb1-984b-ddbb39f48c51 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 486.458171] env[62677]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-2fb4c659-ed50-4bb1-984b-ddbb39f48c51 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 486.471967] env[62677]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2fb4c659-ed50-4bb1-984b-ddbb39f48c51 None None] Session token - acquiring [ 486.472726] env[62677]: DEBUG networking_nsxv3.common.locking [None req-2fb4c659-ed50-4bb1-984b-ddbb39f48c51 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62677) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 486.472918] env[62677]: DEBUG oslo_concurrency.lockutils [None req-2fb4c659-ed50-4bb1-984b-ddbb39f48c51 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62677) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 486.473044] env[62677]: DEBUG oslo_concurrency.lockutils [None req-2fb4c659-ed50-4bb1-984b-ddbb39f48c51 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62677) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 486.473398] env[62677]: DEBUG oslo_concurrency.lockutils [None req-2fb4c659-ed50-4bb1-984b-ddbb39f48c51 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62677) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 486.482287] env[62677]: DEBUG oslo_concurrency.lockutils [None req-2fb4c659-ed50-4bb1-984b-ddbb39f48c51 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62677) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 486.482538] env[62677]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2fb4c659-ed50-4bb1-984b-ddbb39f48c51 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 486.482726] env[62677]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2fb4c659-ed50-4bb1-984b-ddbb39f48c51 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=62677) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 488.490730] env[62677]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2fb4c659-ed50-4bb1-984b-ddbb39f48c51 None None] Session token - acquiring [ 488.491142] env[62677]: DEBUG networking_nsxv3.common.locking [None req-2fb4c659-ed50-4bb1-984b-ddbb39f48c51 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62677) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 488.491353] env[62677]: DEBUG oslo_concurrency.lockutils [None req-2fb4c659-ed50-4bb1-984b-ddbb39f48c51 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62677) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 488.491479] env[62677]: DEBUG oslo_concurrency.lockutils [None req-2fb4c659-ed50-4bb1-984b-ddbb39f48c51 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62677) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 488.491757] env[62677]: DEBUG oslo_concurrency.lockutils [None req-2fb4c659-ed50-4bb1-984b-ddbb39f48c51 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62677) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 488.500823] env[62677]: DEBUG oslo_concurrency.lockutils [None req-2fb4c659-ed50-4bb1-984b-ddbb39f48c51 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62677) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 488.501054] env[62677]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2fb4c659-ed50-4bb1-984b-ddbb39f48c51 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 488.501190] env[62677]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2fb4c659-ed50-4bb1-984b-ddbb39f48c51 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=62677) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 490.504179] env[62677]: CRITICAL neutron [None req-2fb4c659-ed50-4bb1-984b-ddbb39f48c51 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 490.504179] env[62677]: ERROR neutron Traceback (most recent call last): [ 490.504179] env[62677]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 490.504179] env[62677]: ERROR neutron sys.exit(main()) [ 490.504179] env[62677]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 490.504179] env[62677]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 490.504179] env[62677]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 490.504179] env[62677]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 490.504179] env[62677]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 490.504179] env[62677]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 490.504179] env[62677]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 490.504179] env[62677]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 490.504179] env[62677]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 490.504179] env[62677]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 490.504179] env[62677]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 490.504179] env[62677]: ERROR neutron response = self.get(path=path, params=params) [ 490.504951] env[62677]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 490.504951] env[62677]: ERROR neutron raise RuntimeError(msg, last_err) [ 490.504951] env[62677]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 490.504951] env[62677]: ERROR neutron [ 492.953745] env[62686]: INFO neutron.common.config [-] Logging enabled! [ 492.954147] env[62686]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 492.954260] env[62686]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=62686) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 492.954483] env[62686]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 492.954603] env[62686]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 492.955461] env[62686]: WARNING oslo_policy.policy [None req-36a01fe2-63bb-4490-85c4-bafc8108c7f3 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 492.955667] env[62686]: WARNING oslo_policy.policy [None req-36a01fe2-63bb-4490-85c4-bafc8108c7f3 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 492.956899] env[62686]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-36a01fe2-63bb-4490-85c4-bafc8108c7f3 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 492.970631] env[62686]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-36a01fe2-63bb-4490-85c4-bafc8108c7f3 None None] Session token - acquiring [ 492.971385] env[62686]: DEBUG networking_nsxv3.common.locking [None req-36a01fe2-63bb-4490-85c4-bafc8108c7f3 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62686) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 492.971590] env[62686]: DEBUG oslo_concurrency.lockutils [None req-36a01fe2-63bb-4490-85c4-bafc8108c7f3 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62686) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 492.971716] env[62686]: DEBUG oslo_concurrency.lockutils [None req-36a01fe2-63bb-4490-85c4-bafc8108c7f3 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62686) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 492.972083] env[62686]: DEBUG oslo_concurrency.lockutils [None req-36a01fe2-63bb-4490-85c4-bafc8108c7f3 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62686) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 492.981044] env[62686]: DEBUG oslo_concurrency.lockutils [None req-36a01fe2-63bb-4490-85c4-bafc8108c7f3 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62686) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 492.981282] env[62686]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-36a01fe2-63bb-4490-85c4-bafc8108c7f3 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 492.981431] env[62686]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-36a01fe2-63bb-4490-85c4-bafc8108c7f3 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=62686) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 494.989970] env[62686]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-36a01fe2-63bb-4490-85c4-bafc8108c7f3 None None] Session token - acquiring [ 494.990312] env[62686]: DEBUG networking_nsxv3.common.locking [None req-36a01fe2-63bb-4490-85c4-bafc8108c7f3 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62686) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 494.990544] env[62686]: DEBUG oslo_concurrency.lockutils [None req-36a01fe2-63bb-4490-85c4-bafc8108c7f3 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62686) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 494.990681] env[62686]: DEBUG oslo_concurrency.lockutils [None req-36a01fe2-63bb-4490-85c4-bafc8108c7f3 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62686) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 494.990959] env[62686]: DEBUG oslo_concurrency.lockutils [None req-36a01fe2-63bb-4490-85c4-bafc8108c7f3 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62686) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 494.999969] env[62686]: DEBUG oslo_concurrency.lockutils [None req-36a01fe2-63bb-4490-85c4-bafc8108c7f3 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62686) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 495.000229] env[62686]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-36a01fe2-63bb-4490-85c4-bafc8108c7f3 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 495.000368] env[62686]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-36a01fe2-63bb-4490-85c4-bafc8108c7f3 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=62686) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 497.002730] env[62686]: CRITICAL neutron [None req-36a01fe2-63bb-4490-85c4-bafc8108c7f3 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 497.002730] env[62686]: ERROR neutron Traceback (most recent call last): [ 497.002730] env[62686]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 497.002730] env[62686]: ERROR neutron sys.exit(main()) [ 497.002730] env[62686]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 497.002730] env[62686]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 497.002730] env[62686]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 497.002730] env[62686]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 497.002730] env[62686]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 497.002730] env[62686]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 497.002730] env[62686]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 497.002730] env[62686]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 497.002730] env[62686]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 497.002730] env[62686]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 497.002730] env[62686]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 497.002730] env[62686]: ERROR neutron response = self.get(path=path, params=params) [ 497.003520] env[62686]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 497.003520] env[62686]: ERROR neutron raise RuntimeError(msg, last_err) [ 497.003520] env[62686]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 497.003520] env[62686]: ERROR neutron [ 499.465214] env[62734]: INFO neutron.common.config [-] Logging enabled! [ 499.465573] env[62734]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 499.465702] env[62734]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=62734) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 499.465925] env[62734]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 499.466018] env[62734]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 499.466922] env[62734]: WARNING oslo_policy.policy [None req-87947286-f0b0-4602-bb0d-0640df95736d None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 499.467127] env[62734]: WARNING oslo_policy.policy [None req-87947286-f0b0-4602-bb0d-0640df95736d None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 499.468316] env[62734]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-87947286-f0b0-4602-bb0d-0640df95736d None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 499.482550] env[62734]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-87947286-f0b0-4602-bb0d-0640df95736d None None] Session token - acquiring [ 499.483308] env[62734]: DEBUG networking_nsxv3.common.locking [None req-87947286-f0b0-4602-bb0d-0640df95736d None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62734) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 499.483493] env[62734]: DEBUG oslo_concurrency.lockutils [None req-87947286-f0b0-4602-bb0d-0640df95736d None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62734) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 499.483614] env[62734]: DEBUG oslo_concurrency.lockutils [None req-87947286-f0b0-4602-bb0d-0640df95736d None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62734) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 499.483980] env[62734]: DEBUG oslo_concurrency.lockutils [None req-87947286-f0b0-4602-bb0d-0640df95736d None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62734) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 499.492713] env[62734]: DEBUG oslo_concurrency.lockutils [None req-87947286-f0b0-4602-bb0d-0640df95736d None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62734) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 499.492967] env[62734]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-87947286-f0b0-4602-bb0d-0640df95736d None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 499.493123] env[62734]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-87947286-f0b0-4602-bb0d-0640df95736d None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=62734) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 501.499184] env[62734]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-87947286-f0b0-4602-bb0d-0640df95736d None None] Session token - acquiring [ 501.499566] env[62734]: DEBUG networking_nsxv3.common.locking [None req-87947286-f0b0-4602-bb0d-0640df95736d None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62734) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 501.499635] env[62734]: DEBUG oslo_concurrency.lockutils [None req-87947286-f0b0-4602-bb0d-0640df95736d None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62734) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 501.499750] env[62734]: DEBUG oslo_concurrency.lockutils [None req-87947286-f0b0-4602-bb0d-0640df95736d None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62734) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 501.500034] env[62734]: DEBUG oslo_concurrency.lockutils [None req-87947286-f0b0-4602-bb0d-0640df95736d None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62734) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 501.508837] env[62734]: DEBUG oslo_concurrency.lockutils [None req-87947286-f0b0-4602-bb0d-0640df95736d None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62734) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 501.509099] env[62734]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-87947286-f0b0-4602-bb0d-0640df95736d None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 501.509241] env[62734]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-87947286-f0b0-4602-bb0d-0640df95736d None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=62734) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 503.510436] env[62734]: CRITICAL neutron [None req-87947286-f0b0-4602-bb0d-0640df95736d None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 503.510436] env[62734]: ERROR neutron Traceback (most recent call last): [ 503.510436] env[62734]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 503.510436] env[62734]: ERROR neutron sys.exit(main()) [ 503.510436] env[62734]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 503.510436] env[62734]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 503.510436] env[62734]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 503.510436] env[62734]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 503.510436] env[62734]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 503.510436] env[62734]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 503.510436] env[62734]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 503.510436] env[62734]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 503.510436] env[62734]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 503.510436] env[62734]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 503.510436] env[62734]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 503.510436] env[62734]: ERROR neutron response = self.get(path=path, params=params) [ 503.511266] env[62734]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 503.511266] env[62734]: ERROR neutron raise RuntimeError(msg, last_err) [ 503.511266] env[62734]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 503.511266] env[62734]: ERROR neutron [ 506.531555] env[62967]: INFO neutron.common.config [-] Logging enabled! [ 506.531841] env[62967]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 506.531994] env[62967]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=62967) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 506.532234] env[62967]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 506.532322] env[62967]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 506.533222] env[62967]: WARNING oslo_policy.policy [None req-429e5613-bef1-4c9c-b0f0-328485b1b2db None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 506.533437] env[62967]: WARNING oslo_policy.policy [None req-429e5613-bef1-4c9c-b0f0-328485b1b2db None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 506.534690] env[62967]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-429e5613-bef1-4c9c-b0f0-328485b1b2db None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 506.553079] env[62967]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-429e5613-bef1-4c9c-b0f0-328485b1b2db None None] Session token - acquiring [ 506.553975] env[62967]: DEBUG networking_nsxv3.common.locking [None req-429e5613-bef1-4c9c-b0f0-328485b1b2db None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62967) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 506.554183] env[62967]: DEBUG oslo_concurrency.lockutils [None req-429e5613-bef1-4c9c-b0f0-328485b1b2db None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62967) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 506.554307] env[62967]: DEBUG oslo_concurrency.lockutils [None req-429e5613-bef1-4c9c-b0f0-328485b1b2db None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62967) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 506.554687] env[62967]: DEBUG oslo_concurrency.lockutils [None req-429e5613-bef1-4c9c-b0f0-328485b1b2db None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62967) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 506.564071] env[62967]: DEBUG oslo_concurrency.lockutils [None req-429e5613-bef1-4c9c-b0f0-328485b1b2db None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62967) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 506.564413] env[62967]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-429e5613-bef1-4c9c-b0f0-328485b1b2db None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 506.564564] env[62967]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-429e5613-bef1-4c9c-b0f0-328485b1b2db None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=62967) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 508.570685] env[62967]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-429e5613-bef1-4c9c-b0f0-328485b1b2db None None] Session token - acquiring [ 508.570973] env[62967]: DEBUG networking_nsxv3.common.locking [None req-429e5613-bef1-4c9c-b0f0-328485b1b2db None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62967) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 508.571157] env[62967]: DEBUG oslo_concurrency.lockutils [None req-429e5613-bef1-4c9c-b0f0-328485b1b2db None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62967) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 508.571275] env[62967]: DEBUG oslo_concurrency.lockutils [None req-429e5613-bef1-4c9c-b0f0-328485b1b2db None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62967) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 508.571590] env[62967]: DEBUG oslo_concurrency.lockutils [None req-429e5613-bef1-4c9c-b0f0-328485b1b2db None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62967) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 508.582038] env[62967]: DEBUG oslo_concurrency.lockutils [None req-429e5613-bef1-4c9c-b0f0-328485b1b2db None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62967) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 508.582280] env[62967]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-429e5613-bef1-4c9c-b0f0-328485b1b2db None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 508.582473] env[62967]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-429e5613-bef1-4c9c-b0f0-328485b1b2db None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=62967) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 510.585171] env[62967]: CRITICAL neutron [None req-429e5613-bef1-4c9c-b0f0-328485b1b2db None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 510.585171] env[62967]: ERROR neutron Traceback (most recent call last): [ 510.585171] env[62967]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 510.585171] env[62967]: ERROR neutron sys.exit(main()) [ 510.585171] env[62967]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 510.585171] env[62967]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 510.585171] env[62967]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 510.585171] env[62967]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 510.585171] env[62967]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 510.585171] env[62967]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 510.585171] env[62967]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 510.585171] env[62967]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 510.585171] env[62967]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 510.585171] env[62967]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 510.585171] env[62967]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 510.585171] env[62967]: ERROR neutron response = self.get(path=path, params=params) [ 510.585813] env[62967]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 510.585813] env[62967]: ERROR neutron raise RuntimeError(msg, last_err) [ 510.585813] env[62967]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 510.585813] env[62967]: ERROR neutron [ 513.332580] env[63463]: INFO neutron.common.config [-] Logging enabled! [ 513.332580] env[63463]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 513.332580] env[63463]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=63463) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 513.332580] env[63463]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 513.333678] env[63463]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 513.336101] env[63463]: WARNING oslo_policy.policy [None req-22a519a3-6236-45d0-81a3-ec5c414747fa None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 513.336101] env[63463]: WARNING oslo_policy.policy [None req-22a519a3-6236-45d0-81a3-ec5c414747fa None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 513.336101] env[63463]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-22a519a3-6236-45d0-81a3-ec5c414747fa None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 513.355754] env[63463]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-22a519a3-6236-45d0-81a3-ec5c414747fa None None] Session token - acquiring [ 513.356614] env[63463]: DEBUG networking_nsxv3.common.locking [None req-22a519a3-6236-45d0-81a3-ec5c414747fa None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=63463) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 513.356814] env[63463]: DEBUG oslo_concurrency.lockutils [None req-22a519a3-6236-45d0-81a3-ec5c414747fa None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63463) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 513.356982] env[63463]: DEBUG oslo_concurrency.lockutils [None req-22a519a3-6236-45d0-81a3-ec5c414747fa None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63463) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 513.357318] env[63463]: DEBUG oslo_concurrency.lockutils [None req-22a519a3-6236-45d0-81a3-ec5c414747fa None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63463) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 513.370470] env[63463]: DEBUG oslo_concurrency.lockutils [None req-22a519a3-6236-45d0-81a3-ec5c414747fa None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63463) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 513.370730] env[63463]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-22a519a3-6236-45d0-81a3-ec5c414747fa None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 513.370873] env[63463]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-22a519a3-6236-45d0-81a3-ec5c414747fa None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=63463) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 515.377366] env[63463]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-22a519a3-6236-45d0-81a3-ec5c414747fa None None] Session token - acquiring [ 515.377847] env[63463]: DEBUG networking_nsxv3.common.locking [None req-22a519a3-6236-45d0-81a3-ec5c414747fa None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=63463) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 515.377957] env[63463]: DEBUG oslo_concurrency.lockutils [None req-22a519a3-6236-45d0-81a3-ec5c414747fa None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63463) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 515.378076] env[63463]: DEBUG oslo_concurrency.lockutils [None req-22a519a3-6236-45d0-81a3-ec5c414747fa None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63463) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 515.378354] env[63463]: DEBUG oslo_concurrency.lockutils [None req-22a519a3-6236-45d0-81a3-ec5c414747fa None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63463) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 515.387858] env[63463]: DEBUG oslo_concurrency.lockutils [None req-22a519a3-6236-45d0-81a3-ec5c414747fa None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63463) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 515.388434] env[63463]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-22a519a3-6236-45d0-81a3-ec5c414747fa None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 515.388910] env[63463]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-22a519a3-6236-45d0-81a3-ec5c414747fa None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=63463) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 517.392314] env[63463]: CRITICAL neutron [None req-22a519a3-6236-45d0-81a3-ec5c414747fa None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 517.392314] env[63463]: ERROR neutron Traceback (most recent call last): [ 517.392314] env[63463]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 517.392314] env[63463]: ERROR neutron sys.exit(main()) [ 517.392314] env[63463]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 517.392314] env[63463]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 517.392314] env[63463]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 517.392314] env[63463]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 517.392314] env[63463]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 517.392314] env[63463]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 517.392314] env[63463]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 517.392314] env[63463]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 517.392314] env[63463]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 517.392314] env[63463]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 517.392314] env[63463]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 517.392314] env[63463]: ERROR neutron response = self.get(path=path, params=params) [ 517.392934] env[63463]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 517.392934] env[63463]: ERROR neutron raise RuntimeError(msg, last_err) [ 517.392934] env[63463]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 517.392934] env[63463]: ERROR neutron [ 520.199631] env[63954]: INFO neutron.common.config [-] Logging enabled! [ 520.199901] env[63954]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 520.200289] env[63954]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=63954) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 520.200651] env[63954]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 520.200651] env[63954]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 520.201573] env[63954]: WARNING oslo_policy.policy [None req-1bab6400-625c-4810-b476-9bf6c8721e27 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 520.201833] env[63954]: WARNING oslo_policy.policy [None req-1bab6400-625c-4810-b476-9bf6c8721e27 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 520.203091] env[63954]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-1bab6400-625c-4810-b476-9bf6c8721e27 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 520.224262] env[63954]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1bab6400-625c-4810-b476-9bf6c8721e27 None None] Session token - acquiring [ 520.225127] env[63954]: DEBUG networking_nsxv3.common.locking [None req-1bab6400-625c-4810-b476-9bf6c8721e27 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=63954) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 520.225333] env[63954]: DEBUG oslo_concurrency.lockutils [None req-1bab6400-625c-4810-b476-9bf6c8721e27 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 520.225455] env[63954]: DEBUG oslo_concurrency.lockutils [None req-1bab6400-625c-4810-b476-9bf6c8721e27 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 520.225832] env[63954]: DEBUG oslo_concurrency.lockutils [None req-1bab6400-625c-4810-b476-9bf6c8721e27 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 520.239663] env[63954]: DEBUG oslo_concurrency.lockutils [None req-1bab6400-625c-4810-b476-9bf6c8721e27 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 520.239938] env[63954]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1bab6400-625c-4810-b476-9bf6c8721e27 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 520.240109] env[63954]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1bab6400-625c-4810-b476-9bf6c8721e27 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=63954) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 522.246467] env[63954]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1bab6400-625c-4810-b476-9bf6c8721e27 None None] Session token - acquiring [ 522.246467] env[63954]: DEBUG networking_nsxv3.common.locking [None req-1bab6400-625c-4810-b476-9bf6c8721e27 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=63954) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 522.246704] env[63954]: DEBUG oslo_concurrency.lockutils [None req-1bab6400-625c-4810-b476-9bf6c8721e27 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 522.246790] env[63954]: DEBUG oslo_concurrency.lockutils [None req-1bab6400-625c-4810-b476-9bf6c8721e27 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 522.247090] env[63954]: DEBUG oslo_concurrency.lockutils [None req-1bab6400-625c-4810-b476-9bf6c8721e27 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 522.275097] env[63954]: DEBUG oslo_concurrency.lockutils [None req-1bab6400-625c-4810-b476-9bf6c8721e27 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 522.275394] env[63954]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1bab6400-625c-4810-b476-9bf6c8721e27 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 522.275534] env[63954]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1bab6400-625c-4810-b476-9bf6c8721e27 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=63954) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 524.280044] env[63954]: CRITICAL neutron [None req-1bab6400-625c-4810-b476-9bf6c8721e27 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 524.280044] env[63954]: ERROR neutron Traceback (most recent call last): [ 524.280044] env[63954]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 524.280044] env[63954]: ERROR neutron sys.exit(main()) [ 524.280044] env[63954]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 524.280044] env[63954]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 524.280044] env[63954]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 524.280044] env[63954]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 524.280044] env[63954]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 524.280044] env[63954]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 524.280044] env[63954]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 524.280044] env[63954]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 524.280044] env[63954]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 524.280044] env[63954]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 524.280044] env[63954]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 524.280044] env[63954]: ERROR neutron response = self.get(path=path, params=params) [ 524.281482] env[63954]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 524.281482] env[63954]: ERROR neutron raise RuntimeError(msg, last_err) [ 524.281482] env[63954]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 524.281482] env[63954]: ERROR neutron [ 527.183525] env[64437]: INFO neutron.common.config [-] Logging enabled! [ 527.183795] env[64437]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 527.184142] env[64437]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=64437) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 527.184378] env[64437]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 527.184839] env[64437]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 527.185407] env[64437]: WARNING oslo_policy.policy [None req-8de749b3-71fa-45b5-9fa3-c4793a4d50d6 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 527.185622] env[64437]: WARNING oslo_policy.policy [None req-8de749b3-71fa-45b5-9fa3-c4793a4d50d6 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 527.186902] env[64437]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-8de749b3-71fa-45b5-9fa3-c4793a4d50d6 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 527.201157] env[64437]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8de749b3-71fa-45b5-9fa3-c4793a4d50d6 None None] Session token - acquiring [ 527.202049] env[64437]: DEBUG networking_nsxv3.common.locking [None req-8de749b3-71fa-45b5-9fa3-c4793a4d50d6 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=64437) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 527.202255] env[64437]: DEBUG oslo_concurrency.lockutils [None req-8de749b3-71fa-45b5-9fa3-c4793a4d50d6 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64437) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 527.202380] env[64437]: DEBUG oslo_concurrency.lockutils [None req-8de749b3-71fa-45b5-9fa3-c4793a4d50d6 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64437) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 527.202751] env[64437]: DEBUG oslo_concurrency.lockutils [None req-8de749b3-71fa-45b5-9fa3-c4793a4d50d6 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64437) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 527.221758] env[64437]: DEBUG oslo_concurrency.lockutils [None req-8de749b3-71fa-45b5-9fa3-c4793a4d50d6 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64437) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 527.222050] env[64437]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8de749b3-71fa-45b5-9fa3-c4793a4d50d6 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 527.222246] env[64437]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8de749b3-71fa-45b5-9fa3-c4793a4d50d6 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=64437) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 529.229116] env[64437]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8de749b3-71fa-45b5-9fa3-c4793a4d50d6 None None] Session token - acquiring [ 529.229455] env[64437]: DEBUG networking_nsxv3.common.locking [None req-8de749b3-71fa-45b5-9fa3-c4793a4d50d6 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=64437) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 529.229661] env[64437]: DEBUG oslo_concurrency.lockutils [None req-8de749b3-71fa-45b5-9fa3-c4793a4d50d6 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64437) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 529.229785] env[64437]: DEBUG oslo_concurrency.lockutils [None req-8de749b3-71fa-45b5-9fa3-c4793a4d50d6 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64437) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 529.230061] env[64437]: DEBUG oslo_concurrency.lockutils [None req-8de749b3-71fa-45b5-9fa3-c4793a4d50d6 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64437) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 529.242912] env[64437]: DEBUG oslo_concurrency.lockutils [None req-8de749b3-71fa-45b5-9fa3-c4793a4d50d6 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64437) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 529.243180] env[64437]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8de749b3-71fa-45b5-9fa3-c4793a4d50d6 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 529.243322] env[64437]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8de749b3-71fa-45b5-9fa3-c4793a4d50d6 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=64437) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 531.244937] env[64437]: CRITICAL neutron [None req-8de749b3-71fa-45b5-9fa3-c4793a4d50d6 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 531.244937] env[64437]: ERROR neutron Traceback (most recent call last): [ 531.244937] env[64437]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 531.244937] env[64437]: ERROR neutron sys.exit(main()) [ 531.244937] env[64437]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 531.244937] env[64437]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 531.244937] env[64437]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 531.244937] env[64437]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 531.244937] env[64437]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 531.244937] env[64437]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 531.244937] env[64437]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 531.244937] env[64437]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 531.244937] env[64437]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 531.244937] env[64437]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 531.244937] env[64437]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 531.244937] env[64437]: ERROR neutron response = self.get(path=path, params=params) [ 531.245702] env[64437]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 531.245702] env[64437]: ERROR neutron raise RuntimeError(msg, last_err) [ 531.245702] env[64437]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 531.245702] env[64437]: ERROR neutron [ 534.296175] env[64780]: INFO neutron.common.config [-] Logging enabled! [ 534.296493] env[64780]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 534.296787] env[64780]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=64780) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 534.297040] env[64780]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 534.297152] env[64780]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 534.298027] env[64780]: WARNING oslo_policy.policy [None req-85edc2d9-ba34-4747-ae61-e8129dd83914 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 534.298243] env[64780]: WARNING oslo_policy.policy [None req-85edc2d9-ba34-4747-ae61-e8129dd83914 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 534.299498] env[64780]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-85edc2d9-ba34-4747-ae61-e8129dd83914 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 534.314121] env[64780]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-85edc2d9-ba34-4747-ae61-e8129dd83914 None None] Session token - acquiring [ 534.315014] env[64780]: DEBUG networking_nsxv3.common.locking [None req-85edc2d9-ba34-4747-ae61-e8129dd83914 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=64780) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 534.315231] env[64780]: DEBUG oslo_concurrency.lockutils [None req-85edc2d9-ba34-4747-ae61-e8129dd83914 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64780) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 534.315357] env[64780]: DEBUG oslo_concurrency.lockutils [None req-85edc2d9-ba34-4747-ae61-e8129dd83914 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64780) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 534.315756] env[64780]: DEBUG oslo_concurrency.lockutils [None req-85edc2d9-ba34-4747-ae61-e8129dd83914 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64780) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 534.326025] env[64780]: DEBUG oslo_concurrency.lockutils [None req-85edc2d9-ba34-4747-ae61-e8129dd83914 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64780) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 534.326488] env[64780]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-85edc2d9-ba34-4747-ae61-e8129dd83914 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 534.326750] env[64780]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-85edc2d9-ba34-4747-ae61-e8129dd83914 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=64780) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 536.332937] env[64780]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-85edc2d9-ba34-4747-ae61-e8129dd83914 None None] Session token - acquiring [ 536.333646] env[64780]: DEBUG networking_nsxv3.common.locking [None req-85edc2d9-ba34-4747-ae61-e8129dd83914 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=64780) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 536.333861] env[64780]: DEBUG oslo_concurrency.lockutils [None req-85edc2d9-ba34-4747-ae61-e8129dd83914 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64780) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 536.333988] env[64780]: DEBUG oslo_concurrency.lockutils [None req-85edc2d9-ba34-4747-ae61-e8129dd83914 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64780) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 536.334254] env[64780]: DEBUG oslo_concurrency.lockutils [None req-85edc2d9-ba34-4747-ae61-e8129dd83914 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64780) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 536.343487] env[64780]: DEBUG oslo_concurrency.lockutils [None req-85edc2d9-ba34-4747-ae61-e8129dd83914 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64780) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 536.343759] env[64780]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-85edc2d9-ba34-4747-ae61-e8129dd83914 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 536.343941] env[64780]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-85edc2d9-ba34-4747-ae61-e8129dd83914 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=64780) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 538.344914] env[64780]: CRITICAL neutron [None req-85edc2d9-ba34-4747-ae61-e8129dd83914 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 538.344914] env[64780]: ERROR neutron Traceback (most recent call last): [ 538.344914] env[64780]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 538.344914] env[64780]: ERROR neutron sys.exit(main()) [ 538.344914] env[64780]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 538.344914] env[64780]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 538.344914] env[64780]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 538.344914] env[64780]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 538.344914] env[64780]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 538.344914] env[64780]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 538.344914] env[64780]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 538.344914] env[64780]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 538.344914] env[64780]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 538.344914] env[64780]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 538.344914] env[64780]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 538.344914] env[64780]: ERROR neutron response = self.get(path=path, params=params) [ 538.345574] env[64780]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 538.345574] env[64780]: ERROR neutron raise RuntimeError(msg, last_err) [ 538.345574] env[64780]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 538.345574] env[64780]: ERROR neutron [ 541.216873] env[65145]: INFO neutron.common.config [-] Logging enabled! [ 541.217177] env[65145]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 541.217617] env[65145]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=65145) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 541.217918] env[65145]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 541.218026] env[65145]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 541.219060] env[65145]: WARNING oslo_policy.policy [None req-73abf4ba-00a0-4fc5-ae97-8d65d8a19482 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 541.219354] env[65145]: WARNING oslo_policy.policy [None req-73abf4ba-00a0-4fc5-ae97-8d65d8a19482 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 541.221265] env[65145]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-73abf4ba-00a0-4fc5-ae97-8d65d8a19482 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 541.236459] env[65145]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-73abf4ba-00a0-4fc5-ae97-8d65d8a19482 None None] Session token - acquiring [ 541.237607] env[65145]: DEBUG networking_nsxv3.common.locking [None req-73abf4ba-00a0-4fc5-ae97-8d65d8a19482 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=65145) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 541.237607] env[65145]: DEBUG oslo_concurrency.lockutils [None req-73abf4ba-00a0-4fc5-ae97-8d65d8a19482 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65145) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 541.237782] env[65145]: DEBUG oslo_concurrency.lockutils [None req-73abf4ba-00a0-4fc5-ae97-8d65d8a19482 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65145) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 541.237994] env[65145]: DEBUG oslo_concurrency.lockutils [None req-73abf4ba-00a0-4fc5-ae97-8d65d8a19482 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65145) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 541.249887] env[65145]: DEBUG oslo_concurrency.lockutils [None req-73abf4ba-00a0-4fc5-ae97-8d65d8a19482 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65145) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 541.249964] env[65145]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-73abf4ba-00a0-4fc5-ae97-8d65d8a19482 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 541.250276] env[65145]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-73abf4ba-00a0-4fc5-ae97-8d65d8a19482 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=65145) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 543.256813] env[65145]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-73abf4ba-00a0-4fc5-ae97-8d65d8a19482 None None] Session token - acquiring [ 543.257225] env[65145]: DEBUG networking_nsxv3.common.locking [None req-73abf4ba-00a0-4fc5-ae97-8d65d8a19482 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=65145) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 543.257352] env[65145]: DEBUG oslo_concurrency.lockutils [None req-73abf4ba-00a0-4fc5-ae97-8d65d8a19482 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65145) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 543.257483] env[65145]: DEBUG oslo_concurrency.lockutils [None req-73abf4ba-00a0-4fc5-ae97-8d65d8a19482 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65145) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 543.257843] env[65145]: DEBUG oslo_concurrency.lockutils [None req-73abf4ba-00a0-4fc5-ae97-8d65d8a19482 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65145) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 543.268766] env[65145]: DEBUG oslo_concurrency.lockutils [None req-73abf4ba-00a0-4fc5-ae97-8d65d8a19482 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65145) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 543.269078] env[65145]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-73abf4ba-00a0-4fc5-ae97-8d65d8a19482 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 543.269256] env[65145]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-73abf4ba-00a0-4fc5-ae97-8d65d8a19482 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=65145) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 545.272274] env[65145]: CRITICAL neutron [None req-73abf4ba-00a0-4fc5-ae97-8d65d8a19482 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 545.272274] env[65145]: ERROR neutron Traceback (most recent call last): [ 545.272274] env[65145]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 545.272274] env[65145]: ERROR neutron sys.exit(main()) [ 545.272274] env[65145]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 545.272274] env[65145]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 545.272274] env[65145]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 545.272274] env[65145]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 545.272274] env[65145]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 545.272274] env[65145]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 545.272274] env[65145]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 545.272274] env[65145]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 545.272274] env[65145]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 545.272274] env[65145]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 545.272274] env[65145]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 545.272274] env[65145]: ERROR neutron response = self.get(path=path, params=params) [ 545.273050] env[65145]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 545.273050] env[65145]: ERROR neutron raise RuntimeError(msg, last_err) [ 545.273050] env[65145]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 545.273050] env[65145]: ERROR neutron [ 548.232872] env[65413]: INFO neutron.common.config [-] Logging enabled! [ 548.232872] env[65413]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 548.232872] env[65413]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=65413) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 548.232872] env[65413]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 548.232872] env[65413]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 548.233363] env[65413]: WARNING oslo_policy.policy [None req-046a422a-a3f4-4d1f-859a-bcba98de3992 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 548.233399] env[65413]: WARNING oslo_policy.policy [None req-046a422a-a3f4-4d1f-859a-bcba98de3992 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 548.234747] env[65413]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-046a422a-a3f4-4d1f-859a-bcba98de3992 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 548.253242] env[65413]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-046a422a-a3f4-4d1f-859a-bcba98de3992 None None] Session token - acquiring [ 548.254165] env[65413]: DEBUG networking_nsxv3.common.locking [None req-046a422a-a3f4-4d1f-859a-bcba98de3992 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=65413) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 548.254407] env[65413]: DEBUG oslo_concurrency.lockutils [None req-046a422a-a3f4-4d1f-859a-bcba98de3992 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65413) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 548.254501] env[65413]: DEBUG oslo_concurrency.lockutils [None req-046a422a-a3f4-4d1f-859a-bcba98de3992 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65413) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 548.254883] env[65413]: DEBUG oslo_concurrency.lockutils [None req-046a422a-a3f4-4d1f-859a-bcba98de3992 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65413) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 548.266705] env[65413]: DEBUG oslo_concurrency.lockutils [None req-046a422a-a3f4-4d1f-859a-bcba98de3992 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65413) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 548.266705] env[65413]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-046a422a-a3f4-4d1f-859a-bcba98de3992 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 548.266705] env[65413]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-046a422a-a3f4-4d1f-859a-bcba98de3992 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=65413) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 550.272844] env[65413]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-046a422a-a3f4-4d1f-859a-bcba98de3992 None None] Session token - acquiring [ 550.273228] env[65413]: DEBUG networking_nsxv3.common.locking [None req-046a422a-a3f4-4d1f-859a-bcba98de3992 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=65413) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 550.273424] env[65413]: DEBUG oslo_concurrency.lockutils [None req-046a422a-a3f4-4d1f-859a-bcba98de3992 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65413) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 550.273553] env[65413]: DEBUG oslo_concurrency.lockutils [None req-046a422a-a3f4-4d1f-859a-bcba98de3992 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65413) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 550.273952] env[65413]: DEBUG oslo_concurrency.lockutils [None req-046a422a-a3f4-4d1f-859a-bcba98de3992 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65413) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 550.285944] env[65413]: DEBUG oslo_concurrency.lockutils [None req-046a422a-a3f4-4d1f-859a-bcba98de3992 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65413) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 550.286254] env[65413]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-046a422a-a3f4-4d1f-859a-bcba98de3992 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 550.286429] env[65413]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-046a422a-a3f4-4d1f-859a-bcba98de3992 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=65413) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 552.288881] env[65413]: CRITICAL neutron [None req-046a422a-a3f4-4d1f-859a-bcba98de3992 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 552.288881] env[65413]: ERROR neutron Traceback (most recent call last): [ 552.288881] env[65413]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 552.288881] env[65413]: ERROR neutron sys.exit(main()) [ 552.288881] env[65413]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 552.288881] env[65413]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 552.288881] env[65413]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 552.288881] env[65413]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 552.288881] env[65413]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 552.288881] env[65413]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 552.288881] env[65413]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 552.288881] env[65413]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 552.288881] env[65413]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 552.288881] env[65413]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 552.288881] env[65413]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 552.288881] env[65413]: ERROR neutron response = self.get(path=path, params=params) [ 552.289553] env[65413]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 552.289553] env[65413]: ERROR neutron raise RuntimeError(msg, last_err) [ 552.289553] env[65413]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 552.289553] env[65413]: ERROR neutron [ 555.151296] env[65954]: INFO neutron.common.config [-] Logging enabled! [ 555.151614] env[65954]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 555.152174] env[65954]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=65954) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 555.152333] env[65954]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 555.152689] env[65954]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 555.153298] env[65954]: WARNING oslo_policy.policy [None req-e1f191e3-8bc6-48e8-acd8-44b74c298486 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 555.153529] env[65954]: WARNING oslo_policy.policy [None req-e1f191e3-8bc6-48e8-acd8-44b74c298486 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 555.155772] env[65954]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-e1f191e3-8bc6-48e8-acd8-44b74c298486 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 555.170913] env[65954]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e1f191e3-8bc6-48e8-acd8-44b74c298486 None None] Session token - acquiring [ 555.171953] env[65954]: DEBUG networking_nsxv3.common.locking [None req-e1f191e3-8bc6-48e8-acd8-44b74c298486 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=65954) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 555.172042] env[65954]: DEBUG oslo_concurrency.lockutils [None req-e1f191e3-8bc6-48e8-acd8-44b74c298486 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 555.172166] env[65954]: DEBUG oslo_concurrency.lockutils [None req-e1f191e3-8bc6-48e8-acd8-44b74c298486 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 555.172518] env[65954]: DEBUG oslo_concurrency.lockutils [None req-e1f191e3-8bc6-48e8-acd8-44b74c298486 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 555.182411] env[65954]: DEBUG oslo_concurrency.lockutils [None req-e1f191e3-8bc6-48e8-acd8-44b74c298486 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 555.182690] env[65954]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e1f191e3-8bc6-48e8-acd8-44b74c298486 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 555.182846] env[65954]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e1f191e3-8bc6-48e8-acd8-44b74c298486 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=65954) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 557.189022] env[65954]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e1f191e3-8bc6-48e8-acd8-44b74c298486 None None] Session token - acquiring [ 557.189525] env[65954]: DEBUG networking_nsxv3.common.locking [None req-e1f191e3-8bc6-48e8-acd8-44b74c298486 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=65954) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 557.189603] env[65954]: DEBUG oslo_concurrency.lockutils [None req-e1f191e3-8bc6-48e8-acd8-44b74c298486 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 557.189719] env[65954]: DEBUG oslo_concurrency.lockutils [None req-e1f191e3-8bc6-48e8-acd8-44b74c298486 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 557.189993] env[65954]: DEBUG oslo_concurrency.lockutils [None req-e1f191e3-8bc6-48e8-acd8-44b74c298486 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 557.208145] env[65954]: DEBUG oslo_concurrency.lockutils [None req-e1f191e3-8bc6-48e8-acd8-44b74c298486 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 557.208462] env[65954]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e1f191e3-8bc6-48e8-acd8-44b74c298486 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 557.208608] env[65954]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e1f191e3-8bc6-48e8-acd8-44b74c298486 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=65954) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 559.211633] env[65954]: CRITICAL neutron [None req-e1f191e3-8bc6-48e8-acd8-44b74c298486 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 559.211633] env[65954]: ERROR neutron Traceback (most recent call last): [ 559.211633] env[65954]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 559.211633] env[65954]: ERROR neutron sys.exit(main()) [ 559.211633] env[65954]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 559.211633] env[65954]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 559.211633] env[65954]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 559.211633] env[65954]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 559.211633] env[65954]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 559.211633] env[65954]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 559.211633] env[65954]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 559.211633] env[65954]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 559.211633] env[65954]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 559.211633] env[65954]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 559.211633] env[65954]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 559.211633] env[65954]: ERROR neutron response = self.get(path=path, params=params) [ 559.212438] env[65954]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 559.212438] env[65954]: ERROR neutron raise RuntimeError(msg, last_err) [ 559.212438] env[65954]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 559.212438] env[65954]: ERROR neutron [ 562.182020] env[66451]: INFO neutron.common.config [-] Logging enabled! [ 562.182326] env[66451]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 562.182534] env[66451]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=66451) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 562.182757] env[66451]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 562.182850] env[66451]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 562.183707] env[66451]: WARNING oslo_policy.policy [None req-e63ff8bd-3ab9-4201-b46c-7708593af2ac None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 562.183916] env[66451]: WARNING oslo_policy.policy [None req-e63ff8bd-3ab9-4201-b46c-7708593af2ac None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 562.185161] env[66451]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-e63ff8bd-3ab9-4201-b46c-7708593af2ac None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 562.208317] env[66451]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e63ff8bd-3ab9-4201-b46c-7708593af2ac None None] Session token - acquiring [ 562.209217] env[66451]: DEBUG networking_nsxv3.common.locking [None req-e63ff8bd-3ab9-4201-b46c-7708593af2ac None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=66451) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 562.209487] env[66451]: DEBUG oslo_concurrency.lockutils [None req-e63ff8bd-3ab9-4201-b46c-7708593af2ac None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66451) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 562.209549] env[66451]: DEBUG oslo_concurrency.lockutils [None req-e63ff8bd-3ab9-4201-b46c-7708593af2ac None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66451) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 562.209925] env[66451]: DEBUG oslo_concurrency.lockutils [None req-e63ff8bd-3ab9-4201-b46c-7708593af2ac None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66451) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 562.248964] env[66451]: DEBUG oslo_concurrency.lockutils [None req-e63ff8bd-3ab9-4201-b46c-7708593af2ac None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66451) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 562.248964] env[66451]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e63ff8bd-3ab9-4201-b46c-7708593af2ac None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 562.248964] env[66451]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e63ff8bd-3ab9-4201-b46c-7708593af2ac None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=66451) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 564.255729] env[66451]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e63ff8bd-3ab9-4201-b46c-7708593af2ac None None] Session token - acquiring [ 564.256104] env[66451]: DEBUG networking_nsxv3.common.locking [None req-e63ff8bd-3ab9-4201-b46c-7708593af2ac None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=66451) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 564.256325] env[66451]: DEBUG oslo_concurrency.lockutils [None req-e63ff8bd-3ab9-4201-b46c-7708593af2ac None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66451) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 564.256468] env[66451]: DEBUG oslo_concurrency.lockutils [None req-e63ff8bd-3ab9-4201-b46c-7708593af2ac None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66451) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 564.256743] env[66451]: DEBUG oslo_concurrency.lockutils [None req-e63ff8bd-3ab9-4201-b46c-7708593af2ac None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66451) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 564.268549] env[66451]: DEBUG oslo_concurrency.lockutils [None req-e63ff8bd-3ab9-4201-b46c-7708593af2ac None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66451) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 564.268549] env[66451]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e63ff8bd-3ab9-4201-b46c-7708593af2ac None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 564.268549] env[66451]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e63ff8bd-3ab9-4201-b46c-7708593af2ac None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=66451) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 566.268869] env[66451]: CRITICAL neutron [None req-e63ff8bd-3ab9-4201-b46c-7708593af2ac None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 566.268869] env[66451]: ERROR neutron Traceback (most recent call last): [ 566.268869] env[66451]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 566.268869] env[66451]: ERROR neutron sys.exit(main()) [ 566.268869] env[66451]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 566.268869] env[66451]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 566.268869] env[66451]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 566.268869] env[66451]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 566.268869] env[66451]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 566.268869] env[66451]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 566.268869] env[66451]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 566.268869] env[66451]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 566.268869] env[66451]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 566.268869] env[66451]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 566.268869] env[66451]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 566.268869] env[66451]: ERROR neutron response = self.get(path=path, params=params) [ 566.269502] env[66451]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 566.269502] env[66451]: ERROR neutron raise RuntimeError(msg, last_err) [ 566.269502] env[66451]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 566.269502] env[66451]: ERROR neutron [ 569.245620] env[67008]: INFO neutron.common.config [-] Logging enabled! [ 569.245620] env[67008]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 569.245620] env[67008]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=67008) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 569.245620] env[67008]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 569.245959] env[67008]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 569.250138] env[67008]: WARNING oslo_policy.policy [None req-4401eb99-4799-441f-b667-8f9190242fe7 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 569.250138] env[67008]: WARNING oslo_policy.policy [None req-4401eb99-4799-441f-b667-8f9190242fe7 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 569.250138] env[67008]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-4401eb99-4799-441f-b667-8f9190242fe7 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 569.274750] env[67008]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4401eb99-4799-441f-b667-8f9190242fe7 None None] Session token - acquiring [ 569.275695] env[67008]: DEBUG networking_nsxv3.common.locking [None req-4401eb99-4799-441f-b667-8f9190242fe7 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=67008) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 569.276000] env[67008]: DEBUG oslo_concurrency.lockutils [None req-4401eb99-4799-441f-b667-8f9190242fe7 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67008) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 569.276073] env[67008]: DEBUG oslo_concurrency.lockutils [None req-4401eb99-4799-441f-b667-8f9190242fe7 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67008) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 569.276904] env[67008]: DEBUG oslo_concurrency.lockutils [None req-4401eb99-4799-441f-b667-8f9190242fe7 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67008) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 569.286371] env[67008]: DEBUG oslo_concurrency.lockutils [None req-4401eb99-4799-441f-b667-8f9190242fe7 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67008) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 569.286635] env[67008]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4401eb99-4799-441f-b667-8f9190242fe7 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 569.286786] env[67008]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4401eb99-4799-441f-b667-8f9190242fe7 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=67008) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 571.292900] env[67008]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4401eb99-4799-441f-b667-8f9190242fe7 None None] Session token - acquiring [ 571.293286] env[67008]: DEBUG networking_nsxv3.common.locking [None req-4401eb99-4799-441f-b667-8f9190242fe7 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=67008) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 571.293500] env[67008]: DEBUG oslo_concurrency.lockutils [None req-4401eb99-4799-441f-b667-8f9190242fe7 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67008) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 571.293626] env[67008]: DEBUG oslo_concurrency.lockutils [None req-4401eb99-4799-441f-b667-8f9190242fe7 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67008) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 571.293903] env[67008]: DEBUG oslo_concurrency.lockutils [None req-4401eb99-4799-441f-b667-8f9190242fe7 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67008) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 571.303112] env[67008]: DEBUG oslo_concurrency.lockutils [None req-4401eb99-4799-441f-b667-8f9190242fe7 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67008) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 571.303357] env[67008]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4401eb99-4799-441f-b667-8f9190242fe7 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 571.303495] env[67008]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4401eb99-4799-441f-b667-8f9190242fe7 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=67008) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 573.304946] env[67008]: CRITICAL neutron [None req-4401eb99-4799-441f-b667-8f9190242fe7 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 573.304946] env[67008]: ERROR neutron Traceback (most recent call last): [ 573.304946] env[67008]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 573.304946] env[67008]: ERROR neutron sys.exit(main()) [ 573.304946] env[67008]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 573.304946] env[67008]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 573.304946] env[67008]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 573.304946] env[67008]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 573.304946] env[67008]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 573.304946] env[67008]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 573.304946] env[67008]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 573.304946] env[67008]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 573.304946] env[67008]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 573.304946] env[67008]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 573.304946] env[67008]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 573.304946] env[67008]: ERROR neutron response = self.get(path=path, params=params) [ 573.305747] env[67008]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 573.305747] env[67008]: ERROR neutron raise RuntimeError(msg, last_err) [ 573.305747] env[67008]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 573.305747] env[67008]: ERROR neutron [ 576.240432] env[67466]: INFO neutron.common.config [-] Logging enabled! [ 576.240680] env[67466]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 576.241014] env[67466]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=67466) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 576.241231] env[67466]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 576.241320] env[67466]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 576.242187] env[67466]: WARNING oslo_policy.policy [None req-d544ab8a-a46a-47dd-b0bf-80854dd1c7e5 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 576.242500] env[67466]: WARNING oslo_policy.policy [None req-d544ab8a-a46a-47dd-b0bf-80854dd1c7e5 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 576.243856] env[67466]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-d544ab8a-a46a-47dd-b0bf-80854dd1c7e5 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 576.265092] env[67466]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d544ab8a-a46a-47dd-b0bf-80854dd1c7e5 None None] Session token - acquiring [ 576.265964] env[67466]: DEBUG networking_nsxv3.common.locking [None req-d544ab8a-a46a-47dd-b0bf-80854dd1c7e5 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=67466) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 576.266185] env[67466]: DEBUG oslo_concurrency.lockutils [None req-d544ab8a-a46a-47dd-b0bf-80854dd1c7e5 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67466) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 576.266306] env[67466]: DEBUG oslo_concurrency.lockutils [None req-d544ab8a-a46a-47dd-b0bf-80854dd1c7e5 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67466) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 576.266676] env[67466]: DEBUG oslo_concurrency.lockutils [None req-d544ab8a-a46a-47dd-b0bf-80854dd1c7e5 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67466) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 576.275779] env[67466]: DEBUG oslo_concurrency.lockutils [None req-d544ab8a-a46a-47dd-b0bf-80854dd1c7e5 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67466) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 576.276115] env[67466]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d544ab8a-a46a-47dd-b0bf-80854dd1c7e5 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 576.276273] env[67466]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d544ab8a-a46a-47dd-b0bf-80854dd1c7e5 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=67466) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 578.282275] env[67466]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d544ab8a-a46a-47dd-b0bf-80854dd1c7e5 None None] Session token - acquiring [ 578.282690] env[67466]: DEBUG networking_nsxv3.common.locking [None req-d544ab8a-a46a-47dd-b0bf-80854dd1c7e5 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=67466) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 578.282876] env[67466]: DEBUG oslo_concurrency.lockutils [None req-d544ab8a-a46a-47dd-b0bf-80854dd1c7e5 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67466) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 578.283101] env[67466]: DEBUG oslo_concurrency.lockutils [None req-d544ab8a-a46a-47dd-b0bf-80854dd1c7e5 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67466) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 578.283276] env[67466]: DEBUG oslo_concurrency.lockutils [None req-d544ab8a-a46a-47dd-b0bf-80854dd1c7e5 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67466) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 578.292652] env[67466]: DEBUG oslo_concurrency.lockutils [None req-d544ab8a-a46a-47dd-b0bf-80854dd1c7e5 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67466) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 578.292927] env[67466]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d544ab8a-a46a-47dd-b0bf-80854dd1c7e5 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 578.293074] env[67466]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d544ab8a-a46a-47dd-b0bf-80854dd1c7e5 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=67466) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 580.296047] env[67466]: CRITICAL neutron [None req-d544ab8a-a46a-47dd-b0bf-80854dd1c7e5 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 580.296047] env[67466]: ERROR neutron Traceback (most recent call last): [ 580.296047] env[67466]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 580.296047] env[67466]: ERROR neutron sys.exit(main()) [ 580.296047] env[67466]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 580.296047] env[67466]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 580.296047] env[67466]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 580.296047] env[67466]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 580.296047] env[67466]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 580.296047] env[67466]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 580.296047] env[67466]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 580.296047] env[67466]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 580.296047] env[67466]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 580.296047] env[67466]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 580.296047] env[67466]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 580.296047] env[67466]: ERROR neutron response = self.get(path=path, params=params) [ 580.296889] env[67466]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 580.296889] env[67466]: ERROR neutron raise RuntimeError(msg, last_err) [ 580.296889] env[67466]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 580.296889] env[67466]: ERROR neutron [ 583.021630] env[67820]: INFO neutron.common.config [-] Logging enabled! [ 583.021976] env[67820]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 583.022169] env[67820]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=67820) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 583.022392] env[67820]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 583.022483] env[67820]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 583.023492] env[67820]: WARNING oslo_policy.policy [None req-d06449f7-16b0-425d-ad6e-da07ed3afe3e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 583.023585] env[67820]: WARNING oslo_policy.policy [None req-d06449f7-16b0-425d-ad6e-da07ed3afe3e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 583.024871] env[67820]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-d06449f7-16b0-425d-ad6e-da07ed3afe3e None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 583.038932] env[67820]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d06449f7-16b0-425d-ad6e-da07ed3afe3e None None] Session token - acquiring [ 583.039771] env[67820]: DEBUG networking_nsxv3.common.locking [None req-d06449f7-16b0-425d-ad6e-da07ed3afe3e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=67820) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 583.039975] env[67820]: DEBUG oslo_concurrency.lockutils [None req-d06449f7-16b0-425d-ad6e-da07ed3afe3e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67820) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 583.040117] env[67820]: DEBUG oslo_concurrency.lockutils [None req-d06449f7-16b0-425d-ad6e-da07ed3afe3e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67820) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 583.040487] env[67820]: DEBUG oslo_concurrency.lockutils [None req-d06449f7-16b0-425d-ad6e-da07ed3afe3e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67820) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 583.056216] env[67820]: DEBUG oslo_concurrency.lockutils [None req-d06449f7-16b0-425d-ad6e-da07ed3afe3e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67820) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 583.056548] env[67820]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d06449f7-16b0-425d-ad6e-da07ed3afe3e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 583.056703] env[67820]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d06449f7-16b0-425d-ad6e-da07ed3afe3e None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=67820) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 585.063145] env[67820]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d06449f7-16b0-425d-ad6e-da07ed3afe3e None None] Session token - acquiring [ 585.063525] env[67820]: DEBUG networking_nsxv3.common.locking [None req-d06449f7-16b0-425d-ad6e-da07ed3afe3e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=67820) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 585.063625] env[67820]: DEBUG oslo_concurrency.lockutils [None req-d06449f7-16b0-425d-ad6e-da07ed3afe3e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67820) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 585.063747] env[67820]: DEBUG oslo_concurrency.lockutils [None req-d06449f7-16b0-425d-ad6e-da07ed3afe3e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67820) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 585.064035] env[67820]: DEBUG oslo_concurrency.lockutils [None req-d06449f7-16b0-425d-ad6e-da07ed3afe3e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67820) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 585.073301] env[67820]: DEBUG oslo_concurrency.lockutils [None req-d06449f7-16b0-425d-ad6e-da07ed3afe3e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67820) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 585.073540] env[67820]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d06449f7-16b0-425d-ad6e-da07ed3afe3e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 585.073674] env[67820]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d06449f7-16b0-425d-ad6e-da07ed3afe3e None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=67820) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 587.076612] env[67820]: CRITICAL neutron [None req-d06449f7-16b0-425d-ad6e-da07ed3afe3e None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 587.076612] env[67820]: ERROR neutron Traceback (most recent call last): [ 587.076612] env[67820]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 587.076612] env[67820]: ERROR neutron sys.exit(main()) [ 587.076612] env[67820]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 587.076612] env[67820]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 587.076612] env[67820]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 587.076612] env[67820]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 587.076612] env[67820]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 587.076612] env[67820]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 587.076612] env[67820]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 587.076612] env[67820]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 587.076612] env[67820]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 587.076612] env[67820]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 587.076612] env[67820]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 587.076612] env[67820]: ERROR neutron response = self.get(path=path, params=params) [ 587.077456] env[67820]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 587.077456] env[67820]: ERROR neutron raise RuntimeError(msg, last_err) [ 587.077456] env[67820]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 587.077456] env[67820]: ERROR neutron [ 589.875393] env[68103]: INFO neutron.common.config [-] Logging enabled! [ 589.875638] env[68103]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 589.875949] env[68103]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=68103) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 589.876248] env[68103]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 589.876346] env[68103]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 589.877225] env[68103]: WARNING oslo_policy.policy [None req-30dff2cb-9bc8-4e27-bad5-c339ee7e547f None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 589.877471] env[68103]: WARNING oslo_policy.policy [None req-30dff2cb-9bc8-4e27-bad5-c339ee7e547f None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 589.878750] env[68103]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-30dff2cb-9bc8-4e27-bad5-c339ee7e547f None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 589.893710] env[68103]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-30dff2cb-9bc8-4e27-bad5-c339ee7e547f None None] Session token - acquiring [ 589.894469] env[68103]: DEBUG networking_nsxv3.common.locking [None req-30dff2cb-9bc8-4e27-bad5-c339ee7e547f None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68103) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 589.894730] env[68103]: DEBUG oslo_concurrency.lockutils [None req-30dff2cb-9bc8-4e27-bad5-c339ee7e547f None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68103) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 589.894894] env[68103]: DEBUG oslo_concurrency.lockutils [None req-30dff2cb-9bc8-4e27-bad5-c339ee7e547f None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68103) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 589.895325] env[68103]: DEBUG oslo_concurrency.lockutils [None req-30dff2cb-9bc8-4e27-bad5-c339ee7e547f None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68103) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 589.905134] env[68103]: DEBUG oslo_concurrency.lockutils [None req-30dff2cb-9bc8-4e27-bad5-c339ee7e547f None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68103) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 589.905429] env[68103]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-30dff2cb-9bc8-4e27-bad5-c339ee7e547f None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 589.905614] env[68103]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-30dff2cb-9bc8-4e27-bad5-c339ee7e547f None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=68103) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 591.913869] env[68103]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-30dff2cb-9bc8-4e27-bad5-c339ee7e547f None None] Session token - acquiring [ 591.914319] env[68103]: DEBUG networking_nsxv3.common.locking [None req-30dff2cb-9bc8-4e27-bad5-c339ee7e547f None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68103) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 591.914538] env[68103]: DEBUG oslo_concurrency.lockutils [None req-30dff2cb-9bc8-4e27-bad5-c339ee7e547f None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68103) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 591.914666] env[68103]: DEBUG oslo_concurrency.lockutils [None req-30dff2cb-9bc8-4e27-bad5-c339ee7e547f None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68103) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 591.915030] env[68103]: DEBUG oslo_concurrency.lockutils [None req-30dff2cb-9bc8-4e27-bad5-c339ee7e547f None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68103) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 591.926134] env[68103]: DEBUG oslo_concurrency.lockutils [None req-30dff2cb-9bc8-4e27-bad5-c339ee7e547f None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68103) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 591.926403] env[68103]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-30dff2cb-9bc8-4e27-bad5-c339ee7e547f None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 591.926787] env[68103]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-30dff2cb-9bc8-4e27-bad5-c339ee7e547f None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=68103) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 593.929722] env[68103]: CRITICAL neutron [None req-30dff2cb-9bc8-4e27-bad5-c339ee7e547f None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 593.929722] env[68103]: ERROR neutron Traceback (most recent call last): [ 593.929722] env[68103]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 593.929722] env[68103]: ERROR neutron sys.exit(main()) [ 593.929722] env[68103]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 593.929722] env[68103]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 593.929722] env[68103]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 593.929722] env[68103]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 593.929722] env[68103]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 593.929722] env[68103]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 593.929722] env[68103]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 593.929722] env[68103]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 593.929722] env[68103]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 593.929722] env[68103]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 593.929722] env[68103]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 593.929722] env[68103]: ERROR neutron response = self.get(path=path, params=params) [ 593.930807] env[68103]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 593.930807] env[68103]: ERROR neutron raise RuntimeError(msg, last_err) [ 593.930807] env[68103]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 593.930807] env[68103]: ERROR neutron [ 596.579174] env[68239]: INFO neutron.common.config [-] Logging enabled! [ 596.579485] env[68239]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 596.579703] env[68239]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=68239) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 596.579921] env[68239]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 596.580028] env[68239]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 596.580884] env[68239]: WARNING oslo_policy.policy [None req-d94f23c2-a25c-4aab-8cb1-7c068a09eae7 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 596.581100] env[68239]: WARNING oslo_policy.policy [None req-d94f23c2-a25c-4aab-8cb1-7c068a09eae7 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 596.582298] env[68239]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-d94f23c2-a25c-4aab-8cb1-7c068a09eae7 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 596.597335] env[68239]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d94f23c2-a25c-4aab-8cb1-7c068a09eae7 None None] Session token - acquiring [ 596.598575] env[68239]: DEBUG networking_nsxv3.common.locking [None req-d94f23c2-a25c-4aab-8cb1-7c068a09eae7 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68239) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 596.598575] env[68239]: DEBUG oslo_concurrency.lockutils [None req-d94f23c2-a25c-4aab-8cb1-7c068a09eae7 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68239) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 596.598719] env[68239]: DEBUG oslo_concurrency.lockutils [None req-d94f23c2-a25c-4aab-8cb1-7c068a09eae7 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68239) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 596.598868] env[68239]: DEBUG oslo_concurrency.lockutils [None req-d94f23c2-a25c-4aab-8cb1-7c068a09eae7 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68239) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 596.608309] env[68239]: DEBUG oslo_concurrency.lockutils [None req-d94f23c2-a25c-4aab-8cb1-7c068a09eae7 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68239) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 596.608543] env[68239]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d94f23c2-a25c-4aab-8cb1-7c068a09eae7 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 596.608702] env[68239]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d94f23c2-a25c-4aab-8cb1-7c068a09eae7 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=68239) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 598.614697] env[68239]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d94f23c2-a25c-4aab-8cb1-7c068a09eae7 None None] Session token - acquiring [ 598.615158] env[68239]: DEBUG networking_nsxv3.common.locking [None req-d94f23c2-a25c-4aab-8cb1-7c068a09eae7 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68239) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 598.615283] env[68239]: DEBUG oslo_concurrency.lockutils [None req-d94f23c2-a25c-4aab-8cb1-7c068a09eae7 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68239) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 598.615482] env[68239]: DEBUG oslo_concurrency.lockutils [None req-d94f23c2-a25c-4aab-8cb1-7c068a09eae7 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68239) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 598.615676] env[68239]: DEBUG oslo_concurrency.lockutils [None req-d94f23c2-a25c-4aab-8cb1-7c068a09eae7 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68239) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 598.624823] env[68239]: DEBUG oslo_concurrency.lockutils [None req-d94f23c2-a25c-4aab-8cb1-7c068a09eae7 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68239) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 598.625116] env[68239]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d94f23c2-a25c-4aab-8cb1-7c068a09eae7 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 598.625250] env[68239]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d94f23c2-a25c-4aab-8cb1-7c068a09eae7 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=68239) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 600.628226] env[68239]: CRITICAL neutron [None req-d94f23c2-a25c-4aab-8cb1-7c068a09eae7 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 600.628226] env[68239]: ERROR neutron Traceback (most recent call last): [ 600.628226] env[68239]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 600.628226] env[68239]: ERROR neutron sys.exit(main()) [ 600.628226] env[68239]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 600.628226] env[68239]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 600.628226] env[68239]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 600.628226] env[68239]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 600.628226] env[68239]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 600.628226] env[68239]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 600.628226] env[68239]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 600.628226] env[68239]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 600.628226] env[68239]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 600.628226] env[68239]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 600.628226] env[68239]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 600.628226] env[68239]: ERROR neutron response = self.get(path=path, params=params) [ 600.628767] env[68239]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 600.628767] env[68239]: ERROR neutron raise RuntimeError(msg, last_err) [ 600.628767] env[68239]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 600.628767] env[68239]: ERROR neutron [ 603.415685] env[68487]: INFO neutron.common.config [-] Logging enabled! [ 603.415685] env[68487]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 603.416190] env[68487]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=68487) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 603.416441] env[68487]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 603.416533] env[68487]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 603.417507] env[68487]: WARNING oslo_policy.policy [None req-6fa036bb-99f7-4716-aca7-66d4e688eab6 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 603.417641] env[68487]: WARNING oslo_policy.policy [None req-6fa036bb-99f7-4716-aca7-66d4e688eab6 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 603.418899] env[68487]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-6fa036bb-99f7-4716-aca7-66d4e688eab6 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 603.433806] env[68487]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6fa036bb-99f7-4716-aca7-66d4e688eab6 None None] Session token - acquiring [ 603.434656] env[68487]: DEBUG networking_nsxv3.common.locking [None req-6fa036bb-99f7-4716-aca7-66d4e688eab6 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68487) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 603.434862] env[68487]: DEBUG oslo_concurrency.lockutils [None req-6fa036bb-99f7-4716-aca7-66d4e688eab6 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68487) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 603.435007] env[68487]: DEBUG oslo_concurrency.lockutils [None req-6fa036bb-99f7-4716-aca7-66d4e688eab6 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68487) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 603.435389] env[68487]: DEBUG oslo_concurrency.lockutils [None req-6fa036bb-99f7-4716-aca7-66d4e688eab6 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68487) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 603.444751] env[68487]: DEBUG oslo_concurrency.lockutils [None req-6fa036bb-99f7-4716-aca7-66d4e688eab6 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68487) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 603.445029] env[68487]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6fa036bb-99f7-4716-aca7-66d4e688eab6 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 603.445180] env[68487]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6fa036bb-99f7-4716-aca7-66d4e688eab6 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=68487) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 605.451139] env[68487]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6fa036bb-99f7-4716-aca7-66d4e688eab6 None None] Session token - acquiring [ 605.451455] env[68487]: DEBUG networking_nsxv3.common.locking [None req-6fa036bb-99f7-4716-aca7-66d4e688eab6 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68487) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 605.451682] env[68487]: DEBUG oslo_concurrency.lockutils [None req-6fa036bb-99f7-4716-aca7-66d4e688eab6 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68487) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 605.451817] env[68487]: DEBUG oslo_concurrency.lockutils [None req-6fa036bb-99f7-4716-aca7-66d4e688eab6 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68487) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 605.452063] env[68487]: DEBUG oslo_concurrency.lockutils [None req-6fa036bb-99f7-4716-aca7-66d4e688eab6 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68487) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 605.461648] env[68487]: DEBUG oslo_concurrency.lockutils [None req-6fa036bb-99f7-4716-aca7-66d4e688eab6 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68487) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 605.461909] env[68487]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6fa036bb-99f7-4716-aca7-66d4e688eab6 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 605.462049] env[68487]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6fa036bb-99f7-4716-aca7-66d4e688eab6 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=68487) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 607.464969] env[68487]: CRITICAL neutron [None req-6fa036bb-99f7-4716-aca7-66d4e688eab6 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 607.464969] env[68487]: ERROR neutron Traceback (most recent call last): [ 607.464969] env[68487]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 607.464969] env[68487]: ERROR neutron sys.exit(main()) [ 607.464969] env[68487]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 607.464969] env[68487]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 607.464969] env[68487]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 607.464969] env[68487]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 607.464969] env[68487]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 607.464969] env[68487]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 607.464969] env[68487]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 607.464969] env[68487]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 607.464969] env[68487]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 607.464969] env[68487]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 607.464969] env[68487]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 607.464969] env[68487]: ERROR neutron response = self.get(path=path, params=params) [ 607.465596] env[68487]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 607.465596] env[68487]: ERROR neutron raise RuntimeError(msg, last_err) [ 607.465596] env[68487]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 607.465596] env[68487]: ERROR neutron [ 610.091013] env[68859]: INFO neutron.common.config [-] Logging enabled! [ 610.091240] env[68859]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 610.091556] env[68859]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=68859) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 610.091782] env[68859]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 610.091877] env[68859]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 610.092732] env[68859]: WARNING oslo_policy.policy [None req-3ca2d249-54c5-42af-b4ad-acfffb8ce174 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 610.092950] env[68859]: WARNING oslo_policy.policy [None req-3ca2d249-54c5-42af-b4ad-acfffb8ce174 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 610.094183] env[68859]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-3ca2d249-54c5-42af-b4ad-acfffb8ce174 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 610.108295] env[68859]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3ca2d249-54c5-42af-b4ad-acfffb8ce174 None None] Session token - acquiring [ 610.109132] env[68859]: DEBUG networking_nsxv3.common.locking [None req-3ca2d249-54c5-42af-b4ad-acfffb8ce174 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68859) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 610.109343] env[68859]: DEBUG oslo_concurrency.lockutils [None req-3ca2d249-54c5-42af-b4ad-acfffb8ce174 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68859) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 610.109469] env[68859]: DEBUG oslo_concurrency.lockutils [None req-3ca2d249-54c5-42af-b4ad-acfffb8ce174 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68859) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 610.109824] env[68859]: DEBUG oslo_concurrency.lockutils [None req-3ca2d249-54c5-42af-b4ad-acfffb8ce174 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68859) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 610.119332] env[68859]: DEBUG oslo_concurrency.lockutils [None req-3ca2d249-54c5-42af-b4ad-acfffb8ce174 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68859) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 610.119591] env[68859]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3ca2d249-54c5-42af-b4ad-acfffb8ce174 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 610.119744] env[68859]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3ca2d249-54c5-42af-b4ad-acfffb8ce174 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=68859) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 612.125876] env[68859]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3ca2d249-54c5-42af-b4ad-acfffb8ce174 None None] Session token - acquiring [ 612.126291] env[68859]: DEBUG networking_nsxv3.common.locking [None req-3ca2d249-54c5-42af-b4ad-acfffb8ce174 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68859) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 612.126509] env[68859]: DEBUG oslo_concurrency.lockutils [None req-3ca2d249-54c5-42af-b4ad-acfffb8ce174 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68859) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 612.126636] env[68859]: DEBUG oslo_concurrency.lockutils [None req-3ca2d249-54c5-42af-b4ad-acfffb8ce174 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68859) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 612.126914] env[68859]: DEBUG oslo_concurrency.lockutils [None req-3ca2d249-54c5-42af-b4ad-acfffb8ce174 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68859) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 612.136647] env[68859]: DEBUG oslo_concurrency.lockutils [None req-3ca2d249-54c5-42af-b4ad-acfffb8ce174 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68859) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 612.141818] env[68859]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3ca2d249-54c5-42af-b4ad-acfffb8ce174 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 612.141818] env[68859]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3ca2d249-54c5-42af-b4ad-acfffb8ce174 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=68859) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 614.140024] env[68859]: CRITICAL neutron [None req-3ca2d249-54c5-42af-b4ad-acfffb8ce174 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 614.140024] env[68859]: ERROR neutron Traceback (most recent call last): [ 614.140024] env[68859]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 614.140024] env[68859]: ERROR neutron sys.exit(main()) [ 614.140024] env[68859]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 614.140024] env[68859]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 614.140024] env[68859]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 614.140024] env[68859]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 614.140024] env[68859]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 614.140024] env[68859]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 614.140024] env[68859]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 614.140024] env[68859]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 614.140024] env[68859]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 614.140024] env[68859]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 614.140024] env[68859]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 614.140024] env[68859]: ERROR neutron response = self.get(path=path, params=params) [ 614.140721] env[68859]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 614.140721] env[68859]: ERROR neutron raise RuntimeError(msg, last_err) [ 614.140721] env[68859]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 614.140721] env[68859]: ERROR neutron [ 616.841131] env[69138]: INFO neutron.common.config [-] Logging enabled! [ 616.841455] env[69138]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 616.841669] env[69138]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=69138) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 616.841919] env[69138]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 616.841975] env[69138]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 616.842833] env[69138]: WARNING oslo_policy.policy [None req-817d16cb-ff38-4017-88fa-f0fe891680db None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 616.843055] env[69138]: WARNING oslo_policy.policy [None req-817d16cb-ff38-4017-88fa-f0fe891680db None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 616.844325] env[69138]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-817d16cb-ff38-4017-88fa-f0fe891680db None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 616.860714] env[69138]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-817d16cb-ff38-4017-88fa-f0fe891680db None None] Session token - acquiring [ 616.861585] env[69138]: DEBUG networking_nsxv3.common.locking [None req-817d16cb-ff38-4017-88fa-f0fe891680db None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69138) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 616.861794] env[69138]: DEBUG oslo_concurrency.lockutils [None req-817d16cb-ff38-4017-88fa-f0fe891680db None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69138) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 616.861921] env[69138]: DEBUG oslo_concurrency.lockutils [None req-817d16cb-ff38-4017-88fa-f0fe891680db None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69138) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 616.862297] env[69138]: DEBUG oslo_concurrency.lockutils [None req-817d16cb-ff38-4017-88fa-f0fe891680db None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69138) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 616.871853] env[69138]: DEBUG oslo_concurrency.lockutils [None req-817d16cb-ff38-4017-88fa-f0fe891680db None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69138) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 616.872132] env[69138]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-817d16cb-ff38-4017-88fa-f0fe891680db None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 616.872288] env[69138]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-817d16cb-ff38-4017-88fa-f0fe891680db None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=69138) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 618.878275] env[69138]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-817d16cb-ff38-4017-88fa-f0fe891680db None None] Session token - acquiring [ 618.878591] env[69138]: DEBUG networking_nsxv3.common.locking [None req-817d16cb-ff38-4017-88fa-f0fe891680db None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69138) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 618.878786] env[69138]: DEBUG oslo_concurrency.lockutils [None req-817d16cb-ff38-4017-88fa-f0fe891680db None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69138) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 618.878910] env[69138]: DEBUG oslo_concurrency.lockutils [None req-817d16cb-ff38-4017-88fa-f0fe891680db None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69138) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 618.879192] env[69138]: DEBUG oslo_concurrency.lockutils [None req-817d16cb-ff38-4017-88fa-f0fe891680db None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69138) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 618.888910] env[69138]: DEBUG oslo_concurrency.lockutils [None req-817d16cb-ff38-4017-88fa-f0fe891680db None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69138) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 618.889161] env[69138]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-817d16cb-ff38-4017-88fa-f0fe891680db None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 618.889297] env[69138]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-817d16cb-ff38-4017-88fa-f0fe891680db None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=69138) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 620.892213] env[69138]: CRITICAL neutron [None req-817d16cb-ff38-4017-88fa-f0fe891680db None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 620.892213] env[69138]: ERROR neutron Traceback (most recent call last): [ 620.892213] env[69138]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 620.892213] env[69138]: ERROR neutron sys.exit(main()) [ 620.892213] env[69138]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 620.892213] env[69138]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 620.892213] env[69138]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 620.892213] env[69138]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 620.892213] env[69138]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 620.892213] env[69138]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 620.892213] env[69138]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 620.892213] env[69138]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 620.892213] env[69138]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 620.892213] env[69138]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 620.892213] env[69138]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 620.892213] env[69138]: ERROR neutron response = self.get(path=path, params=params) [ 620.892868] env[69138]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 620.892868] env[69138]: ERROR neutron raise RuntimeError(msg, last_err) [ 620.892868] env[69138]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 620.892868] env[69138]: ERROR neutron [ 623.641982] env[69225]: INFO neutron.common.config [-] Logging enabled! [ 623.642228] env[69225]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 623.642531] env[69225]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=69225) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 623.642762] env[69225]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 623.642878] env[69225]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 623.643743] env[69225]: WARNING oslo_policy.policy [None req-464f0bde-abe0-4a35-91ca-f60eefd115ce None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 623.643967] env[69225]: WARNING oslo_policy.policy [None req-464f0bde-abe0-4a35-91ca-f60eefd115ce None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 623.645263] env[69225]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-464f0bde-abe0-4a35-91ca-f60eefd115ce None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 623.660166] env[69225]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-464f0bde-abe0-4a35-91ca-f60eefd115ce None None] Session token - acquiring [ 623.661041] env[69225]: DEBUG networking_nsxv3.common.locking [None req-464f0bde-abe0-4a35-91ca-f60eefd115ce None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69225) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 623.661257] env[69225]: DEBUG oslo_concurrency.lockutils [None req-464f0bde-abe0-4a35-91ca-f60eefd115ce None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69225) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 623.661775] env[69225]: DEBUG oslo_concurrency.lockutils [None req-464f0bde-abe0-4a35-91ca-f60eefd115ce None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69225) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 623.661775] env[69225]: DEBUG oslo_concurrency.lockutils [None req-464f0bde-abe0-4a35-91ca-f60eefd115ce None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69225) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 623.671903] env[69225]: DEBUG oslo_concurrency.lockutils [None req-464f0bde-abe0-4a35-91ca-f60eefd115ce None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69225) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 623.672232] env[69225]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-464f0bde-abe0-4a35-91ca-f60eefd115ce None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 623.672402] env[69225]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-464f0bde-abe0-4a35-91ca-f60eefd115ce None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=69225) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 625.678617] env[69225]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-464f0bde-abe0-4a35-91ca-f60eefd115ce None None] Session token - acquiring [ 625.678992] env[69225]: DEBUG networking_nsxv3.common.locking [None req-464f0bde-abe0-4a35-91ca-f60eefd115ce None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69225) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 625.679150] env[69225]: DEBUG oslo_concurrency.lockutils [None req-464f0bde-abe0-4a35-91ca-f60eefd115ce None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69225) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 625.679279] env[69225]: DEBUG oslo_concurrency.lockutils [None req-464f0bde-abe0-4a35-91ca-f60eefd115ce None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69225) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 625.679555] env[69225]: DEBUG oslo_concurrency.lockutils [None req-464f0bde-abe0-4a35-91ca-f60eefd115ce None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69225) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 625.690885] env[69225]: DEBUG oslo_concurrency.lockutils [None req-464f0bde-abe0-4a35-91ca-f60eefd115ce None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69225) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 625.691397] env[69225]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-464f0bde-abe0-4a35-91ca-f60eefd115ce None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 625.691860] env[69225]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-464f0bde-abe0-4a35-91ca-f60eefd115ce None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=69225) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 627.692973] env[69225]: CRITICAL neutron [None req-464f0bde-abe0-4a35-91ca-f60eefd115ce None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 627.692973] env[69225]: ERROR neutron Traceback (most recent call last): [ 627.692973] env[69225]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 627.692973] env[69225]: ERROR neutron sys.exit(main()) [ 627.692973] env[69225]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 627.692973] env[69225]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 627.692973] env[69225]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 627.692973] env[69225]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 627.692973] env[69225]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 627.692973] env[69225]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 627.692973] env[69225]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 627.692973] env[69225]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 627.692973] env[69225]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 627.692973] env[69225]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 627.692973] env[69225]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 627.692973] env[69225]: ERROR neutron response = self.get(path=path, params=params) [ 627.693587] env[69225]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 627.693587] env[69225]: ERROR neutron raise RuntimeError(msg, last_err) [ 627.693587] env[69225]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 627.693587] env[69225]: ERROR neutron [ 630.438055] env[69564]: INFO neutron.common.config [-] Logging enabled! [ 630.438274] env[69564]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 630.438631] env[69564]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=69564) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 630.438879] env[69564]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 630.439019] env[69564]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 630.439955] env[69564]: WARNING oslo_policy.policy [None req-b1968db8-7c4a-48f7-9061-8b58df5437c6 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 630.440196] env[69564]: WARNING oslo_policy.policy [None req-b1968db8-7c4a-48f7-9061-8b58df5437c6 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 630.441418] env[69564]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-b1968db8-7c4a-48f7-9061-8b58df5437c6 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 630.457538] env[69564]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b1968db8-7c4a-48f7-9061-8b58df5437c6 None None] Session token - acquiring [ 630.458380] env[69564]: DEBUG networking_nsxv3.common.locking [None req-b1968db8-7c4a-48f7-9061-8b58df5437c6 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69564) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 630.458582] env[69564]: DEBUG oslo_concurrency.lockutils [None req-b1968db8-7c4a-48f7-9061-8b58df5437c6 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69564) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 630.458709] env[69564]: DEBUG oslo_concurrency.lockutils [None req-b1968db8-7c4a-48f7-9061-8b58df5437c6 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69564) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 630.459197] env[69564]: DEBUG oslo_concurrency.lockutils [None req-b1968db8-7c4a-48f7-9061-8b58df5437c6 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69564) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 630.468377] env[69564]: DEBUG oslo_concurrency.lockutils [None req-b1968db8-7c4a-48f7-9061-8b58df5437c6 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69564) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 630.468642] env[69564]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b1968db8-7c4a-48f7-9061-8b58df5437c6 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 630.468795] env[69564]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b1968db8-7c4a-48f7-9061-8b58df5437c6 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=69564) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 632.475368] env[69564]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b1968db8-7c4a-48f7-9061-8b58df5437c6 None None] Session token - acquiring [ 632.475625] env[69564]: DEBUG networking_nsxv3.common.locking [None req-b1968db8-7c4a-48f7-9061-8b58df5437c6 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69564) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 632.475737] env[69564]: DEBUG oslo_concurrency.lockutils [None req-b1968db8-7c4a-48f7-9061-8b58df5437c6 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69564) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 632.475883] env[69564]: DEBUG oslo_concurrency.lockutils [None req-b1968db8-7c4a-48f7-9061-8b58df5437c6 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69564) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 632.476224] env[69564]: DEBUG oslo_concurrency.lockutils [None req-b1968db8-7c4a-48f7-9061-8b58df5437c6 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69564) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 632.486193] env[69564]: DEBUG oslo_concurrency.lockutils [None req-b1968db8-7c4a-48f7-9061-8b58df5437c6 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69564) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 632.486439] env[69564]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b1968db8-7c4a-48f7-9061-8b58df5437c6 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 632.486582] env[69564]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b1968db8-7c4a-48f7-9061-8b58df5437c6 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=69564) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 634.488915] env[69564]: CRITICAL neutron [None req-b1968db8-7c4a-48f7-9061-8b58df5437c6 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 634.488915] env[69564]: ERROR neutron Traceback (most recent call last): [ 634.488915] env[69564]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 634.488915] env[69564]: ERROR neutron sys.exit(main()) [ 634.488915] env[69564]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 634.488915] env[69564]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 634.488915] env[69564]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 634.488915] env[69564]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 634.488915] env[69564]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 634.488915] env[69564]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 634.488915] env[69564]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 634.488915] env[69564]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 634.488915] env[69564]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 634.488915] env[69564]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 634.488915] env[69564]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 634.488915] env[69564]: ERROR neutron response = self.get(path=path, params=params) [ 634.489572] env[69564]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 634.489572] env[69564]: ERROR neutron raise RuntimeError(msg, last_err) [ 634.489572] env[69564]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 634.489572] env[69564]: ERROR neutron [ 637.096382] env[69789]: INFO neutron.common.config [-] Logging enabled! [ 637.096740] env[69789]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 637.097014] env[69789]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=69789) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 637.097088] env[69789]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 637.097168] env[69789]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 637.098030] env[69789]: WARNING oslo_policy.policy [None req-349154f8-95ff-49aa-898e-e25ce24c4afd None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 637.098246] env[69789]: WARNING oslo_policy.policy [None req-349154f8-95ff-49aa-898e-e25ce24c4afd None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 637.099448] env[69789]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-349154f8-95ff-49aa-898e-e25ce24c4afd None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 637.123412] env[69789]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-349154f8-95ff-49aa-898e-e25ce24c4afd None None] Session token - acquiring [ 637.124326] env[69789]: DEBUG networking_nsxv3.common.locking [None req-349154f8-95ff-49aa-898e-e25ce24c4afd None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69789) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 637.124533] env[69789]: DEBUG oslo_concurrency.lockutils [None req-349154f8-95ff-49aa-898e-e25ce24c4afd None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69789) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 637.124661] env[69789]: DEBUG oslo_concurrency.lockutils [None req-349154f8-95ff-49aa-898e-e25ce24c4afd None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69789) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 637.125108] env[69789]: DEBUG oslo_concurrency.lockutils [None req-349154f8-95ff-49aa-898e-e25ce24c4afd None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69789) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 637.135175] env[69789]: DEBUG oslo_concurrency.lockutils [None req-349154f8-95ff-49aa-898e-e25ce24c4afd None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69789) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 637.135442] env[69789]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-349154f8-95ff-49aa-898e-e25ce24c4afd None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 637.135593] env[69789]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-349154f8-95ff-49aa-898e-e25ce24c4afd None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=69789) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 639.144145] env[69789]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-349154f8-95ff-49aa-898e-e25ce24c4afd None None] Session token - acquiring [ 639.144145] env[69789]: DEBUG networking_nsxv3.common.locking [None req-349154f8-95ff-49aa-898e-e25ce24c4afd None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69789) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 639.144145] env[69789]: DEBUG oslo_concurrency.lockutils [None req-349154f8-95ff-49aa-898e-e25ce24c4afd None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69789) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 639.144521] env[69789]: DEBUG oslo_concurrency.lockutils [None req-349154f8-95ff-49aa-898e-e25ce24c4afd None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69789) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 639.144521] env[69789]: DEBUG oslo_concurrency.lockutils [None req-349154f8-95ff-49aa-898e-e25ce24c4afd None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69789) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 639.152904] env[69789]: DEBUG oslo_concurrency.lockutils [None req-349154f8-95ff-49aa-898e-e25ce24c4afd None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69789) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 639.152904] env[69789]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-349154f8-95ff-49aa-898e-e25ce24c4afd None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 639.152904] env[69789]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-349154f8-95ff-49aa-898e-e25ce24c4afd None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=69789) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 641.155672] env[69789]: CRITICAL neutron [None req-349154f8-95ff-49aa-898e-e25ce24c4afd None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 641.155672] env[69789]: ERROR neutron Traceback (most recent call last): [ 641.155672] env[69789]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 641.155672] env[69789]: ERROR neutron sys.exit(main()) [ 641.155672] env[69789]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 641.155672] env[69789]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 641.155672] env[69789]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 641.155672] env[69789]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 641.155672] env[69789]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 641.155672] env[69789]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 641.155672] env[69789]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 641.155672] env[69789]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 641.155672] env[69789]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 641.155672] env[69789]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 641.155672] env[69789]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 641.155672] env[69789]: ERROR neutron response = self.get(path=path, params=params) [ 641.156487] env[69789]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 641.156487] env[69789]: ERROR neutron raise RuntimeError(msg, last_err) [ 641.156487] env[69789]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 641.156487] env[69789]: ERROR neutron [ 643.852017] env[69867]: INFO neutron.common.config [-] Logging enabled! [ 643.852317] env[69867]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 643.852561] env[69867]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=69867) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 643.852789] env[69867]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 643.852907] env[69867]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 643.853778] env[69867]: WARNING oslo_policy.policy [None req-9815142a-ff47-4414-b38a-1f43b64a232a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 643.853999] env[69867]: WARNING oslo_policy.policy [None req-9815142a-ff47-4414-b38a-1f43b64a232a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 643.855247] env[69867]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-9815142a-ff47-4414-b38a-1f43b64a232a None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 643.869628] env[69867]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9815142a-ff47-4414-b38a-1f43b64a232a None None] Session token - acquiring [ 643.870420] env[69867]: DEBUG networking_nsxv3.common.locking [None req-9815142a-ff47-4414-b38a-1f43b64a232a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69867) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 643.870624] env[69867]: DEBUG oslo_concurrency.lockutils [None req-9815142a-ff47-4414-b38a-1f43b64a232a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69867) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 643.870752] env[69867]: DEBUG oslo_concurrency.lockutils [None req-9815142a-ff47-4414-b38a-1f43b64a232a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69867) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 643.871123] env[69867]: DEBUG oslo_concurrency.lockutils [None req-9815142a-ff47-4414-b38a-1f43b64a232a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69867) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 643.880332] env[69867]: DEBUG oslo_concurrency.lockutils [None req-9815142a-ff47-4414-b38a-1f43b64a232a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69867) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 643.880597] env[69867]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9815142a-ff47-4414-b38a-1f43b64a232a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 643.880746] env[69867]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9815142a-ff47-4414-b38a-1f43b64a232a None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=69867) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 645.887418] env[69867]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9815142a-ff47-4414-b38a-1f43b64a232a None None] Session token - acquiring [ 645.887418] env[69867]: DEBUG networking_nsxv3.common.locking [None req-9815142a-ff47-4414-b38a-1f43b64a232a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69867) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 645.887418] env[69867]: DEBUG oslo_concurrency.lockutils [None req-9815142a-ff47-4414-b38a-1f43b64a232a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69867) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 645.887778] env[69867]: DEBUG oslo_concurrency.lockutils [None req-9815142a-ff47-4414-b38a-1f43b64a232a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69867) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 645.887778] env[69867]: DEBUG oslo_concurrency.lockutils [None req-9815142a-ff47-4414-b38a-1f43b64a232a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69867) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 645.900054] env[69867]: DEBUG oslo_concurrency.lockutils [None req-9815142a-ff47-4414-b38a-1f43b64a232a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69867) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 645.900054] env[69867]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9815142a-ff47-4414-b38a-1f43b64a232a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 645.900054] env[69867]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9815142a-ff47-4414-b38a-1f43b64a232a None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=69867) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 647.903024] env[69867]: CRITICAL neutron [None req-9815142a-ff47-4414-b38a-1f43b64a232a None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 647.903024] env[69867]: ERROR neutron Traceback (most recent call last): [ 647.903024] env[69867]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 647.903024] env[69867]: ERROR neutron sys.exit(main()) [ 647.903024] env[69867]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 647.903024] env[69867]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 647.903024] env[69867]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 647.903024] env[69867]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 647.903024] env[69867]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 647.903024] env[69867]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 647.903024] env[69867]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 647.903024] env[69867]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 647.903024] env[69867]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 647.903024] env[69867]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 647.903024] env[69867]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 647.903024] env[69867]: ERROR neutron response = self.get(path=path, params=params) [ 647.903843] env[69867]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 647.903843] env[69867]: ERROR neutron raise RuntimeError(msg, last_err) [ 647.903843] env[69867]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 647.903843] env[69867]: ERROR neutron [ 650.586920] env[70070]: INFO neutron.common.config [-] Logging enabled! [ 650.587292] env[70070]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 650.587461] env[70070]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=70070) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 650.587693] env[70070]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 650.587800] env[70070]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 650.588648] env[70070]: WARNING oslo_policy.policy [None req-fa742275-b037-49a3-9eb7-3f2cf75356a2 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 650.588905] env[70070]: WARNING oslo_policy.policy [None req-fa742275-b037-49a3-9eb7-3f2cf75356a2 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 650.590088] env[70070]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-fa742275-b037-49a3-9eb7-3f2cf75356a2 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 650.604662] env[70070]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fa742275-b037-49a3-9eb7-3f2cf75356a2 None None] Session token - acquiring [ 650.605515] env[70070]: DEBUG networking_nsxv3.common.locking [None req-fa742275-b037-49a3-9eb7-3f2cf75356a2 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70070) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 650.605718] env[70070]: DEBUG oslo_concurrency.lockutils [None req-fa742275-b037-49a3-9eb7-3f2cf75356a2 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70070) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 650.605847] env[70070]: DEBUG oslo_concurrency.lockutils [None req-fa742275-b037-49a3-9eb7-3f2cf75356a2 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70070) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 650.606201] env[70070]: DEBUG oslo_concurrency.lockutils [None req-fa742275-b037-49a3-9eb7-3f2cf75356a2 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70070) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 650.615625] env[70070]: DEBUG oslo_concurrency.lockutils [None req-fa742275-b037-49a3-9eb7-3f2cf75356a2 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70070) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 650.616340] env[70070]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fa742275-b037-49a3-9eb7-3f2cf75356a2 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 650.616340] env[70070]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fa742275-b037-49a3-9eb7-3f2cf75356a2 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=70070) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 652.622259] env[70070]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fa742275-b037-49a3-9eb7-3f2cf75356a2 None None] Session token - acquiring [ 652.622642] env[70070]: DEBUG networking_nsxv3.common.locking [None req-fa742275-b037-49a3-9eb7-3f2cf75356a2 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70070) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 652.622866] env[70070]: DEBUG oslo_concurrency.lockutils [None req-fa742275-b037-49a3-9eb7-3f2cf75356a2 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70070) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 652.623002] env[70070]: DEBUG oslo_concurrency.lockutils [None req-fa742275-b037-49a3-9eb7-3f2cf75356a2 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70070) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 652.623280] env[70070]: DEBUG oslo_concurrency.lockutils [None req-fa742275-b037-49a3-9eb7-3f2cf75356a2 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70070) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 652.633100] env[70070]: DEBUG oslo_concurrency.lockutils [None req-fa742275-b037-49a3-9eb7-3f2cf75356a2 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70070) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 652.633400] env[70070]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fa742275-b037-49a3-9eb7-3f2cf75356a2 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 652.633609] env[70070]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fa742275-b037-49a3-9eb7-3f2cf75356a2 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=70070) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 654.636642] env[70070]: CRITICAL neutron [None req-fa742275-b037-49a3-9eb7-3f2cf75356a2 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 654.636642] env[70070]: ERROR neutron Traceback (most recent call last): [ 654.636642] env[70070]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 654.636642] env[70070]: ERROR neutron sys.exit(main()) [ 654.636642] env[70070]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 654.636642] env[70070]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 654.636642] env[70070]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 654.636642] env[70070]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 654.636642] env[70070]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 654.636642] env[70070]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 654.636642] env[70070]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 654.636642] env[70070]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 654.636642] env[70070]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 654.636642] env[70070]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 654.636642] env[70070]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 654.636642] env[70070]: ERROR neutron response = self.get(path=path, params=params) [ 654.637360] env[70070]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 654.637360] env[70070]: ERROR neutron raise RuntimeError(msg, last_err) [ 654.637360] env[70070]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 654.637360] env[70070]: ERROR neutron [ 657.529084] env[70143]: INFO neutron.common.config [-] Logging enabled! [ 657.529360] env[70143]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 657.529658] env[70143]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=70143) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 657.530047] env[70143]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 657.530047] env[70143]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 657.530815] env[70143]: WARNING oslo_policy.policy [None req-b01f04df-1fc9-4724-be03-c30464016b91 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 657.531017] env[70143]: WARNING oslo_policy.policy [None req-b01f04df-1fc9-4724-be03-c30464016b91 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 657.532271] env[70143]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-b01f04df-1fc9-4724-be03-c30464016b91 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 657.548048] env[70143]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b01f04df-1fc9-4724-be03-c30464016b91 None None] Session token - acquiring [ 657.548596] env[70143]: DEBUG networking_nsxv3.common.locking [None req-b01f04df-1fc9-4724-be03-c30464016b91 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70143) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 657.548795] env[70143]: DEBUG oslo_concurrency.lockutils [None req-b01f04df-1fc9-4724-be03-c30464016b91 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70143) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 657.548925] env[70143]: DEBUG oslo_concurrency.lockutils [None req-b01f04df-1fc9-4724-be03-c30464016b91 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70143) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 657.549320] env[70143]: DEBUG oslo_concurrency.lockutils [None req-b01f04df-1fc9-4724-be03-c30464016b91 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70143) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 657.558840] env[70143]: DEBUG oslo_concurrency.lockutils [None req-b01f04df-1fc9-4724-be03-c30464016b91 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70143) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 657.559083] env[70143]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b01f04df-1fc9-4724-be03-c30464016b91 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 657.559229] env[70143]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b01f04df-1fc9-4724-be03-c30464016b91 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=70143) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 659.565728] env[70143]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b01f04df-1fc9-4724-be03-c30464016b91 None None] Session token - acquiring [ 659.565728] env[70143]: DEBUG networking_nsxv3.common.locking [None req-b01f04df-1fc9-4724-be03-c30464016b91 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70143) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 659.566240] env[70143]: DEBUG oslo_concurrency.lockutils [None req-b01f04df-1fc9-4724-be03-c30464016b91 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70143) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 659.566240] env[70143]: DEBUG oslo_concurrency.lockutils [None req-b01f04df-1fc9-4724-be03-c30464016b91 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70143) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 659.566435] env[70143]: DEBUG oslo_concurrency.lockutils [None req-b01f04df-1fc9-4724-be03-c30464016b91 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70143) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 659.575854] env[70143]: DEBUG oslo_concurrency.lockutils [None req-b01f04df-1fc9-4724-be03-c30464016b91 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70143) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 659.576205] env[70143]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b01f04df-1fc9-4724-be03-c30464016b91 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 659.576345] env[70143]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b01f04df-1fc9-4724-be03-c30464016b91 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=70143) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 661.579326] env[70143]: CRITICAL neutron [None req-b01f04df-1fc9-4724-be03-c30464016b91 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 661.579326] env[70143]: ERROR neutron Traceback (most recent call last): [ 661.579326] env[70143]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 661.579326] env[70143]: ERROR neutron sys.exit(main()) [ 661.579326] env[70143]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 661.579326] env[70143]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 661.579326] env[70143]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 661.579326] env[70143]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 661.579326] env[70143]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 661.579326] env[70143]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 661.579326] env[70143]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 661.579326] env[70143]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 661.579326] env[70143]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 661.579326] env[70143]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 661.579326] env[70143]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 661.579326] env[70143]: ERROR neutron response = self.get(path=path, params=params) [ 661.580060] env[70143]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 661.580060] env[70143]: ERROR neutron raise RuntimeError(msg, last_err) [ 661.580060] env[70143]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 661.580060] env[70143]: ERROR neutron [ 664.399175] env[70385]: INFO neutron.common.config [-] Logging enabled! [ 664.399492] env[70385]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 664.399717] env[70385]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=70385) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 664.399942] env[70385]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 664.400105] env[70385]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 664.400963] env[70385]: WARNING oslo_policy.policy [None req-ebef5157-3246-4e31-bc48-bad93d3405e6 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 664.401181] env[70385]: WARNING oslo_policy.policy [None req-ebef5157-3246-4e31-bc48-bad93d3405e6 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 664.402407] env[70385]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-ebef5157-3246-4e31-bc48-bad93d3405e6 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 664.416758] env[70385]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ebef5157-3246-4e31-bc48-bad93d3405e6 None None] Session token - acquiring [ 664.417608] env[70385]: DEBUG networking_nsxv3.common.locking [None req-ebef5157-3246-4e31-bc48-bad93d3405e6 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70385) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 664.417800] env[70385]: DEBUG oslo_concurrency.lockutils [None req-ebef5157-3246-4e31-bc48-bad93d3405e6 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70385) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 664.417921] env[70385]: DEBUG oslo_concurrency.lockutils [None req-ebef5157-3246-4e31-bc48-bad93d3405e6 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70385) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 664.418277] env[70385]: DEBUG oslo_concurrency.lockutils [None req-ebef5157-3246-4e31-bc48-bad93d3405e6 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70385) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 664.427945] env[70385]: DEBUG oslo_concurrency.lockutils [None req-ebef5157-3246-4e31-bc48-bad93d3405e6 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70385) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 664.427945] env[70385]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ebef5157-3246-4e31-bc48-bad93d3405e6 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 664.427945] env[70385]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ebef5157-3246-4e31-bc48-bad93d3405e6 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=70385) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 666.434066] env[70385]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ebef5157-3246-4e31-bc48-bad93d3405e6 None None] Session token - acquiring [ 666.434333] env[70385]: DEBUG networking_nsxv3.common.locking [None req-ebef5157-3246-4e31-bc48-bad93d3405e6 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70385) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 666.434333] env[70385]: DEBUG oslo_concurrency.lockutils [None req-ebef5157-3246-4e31-bc48-bad93d3405e6 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70385) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 666.434453] env[70385]: DEBUG oslo_concurrency.lockutils [None req-ebef5157-3246-4e31-bc48-bad93d3405e6 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70385) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 666.434665] env[70385]: DEBUG oslo_concurrency.lockutils [None req-ebef5157-3246-4e31-bc48-bad93d3405e6 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70385) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 666.443952] env[70385]: DEBUG oslo_concurrency.lockutils [None req-ebef5157-3246-4e31-bc48-bad93d3405e6 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70385) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 666.444249] env[70385]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ebef5157-3246-4e31-bc48-bad93d3405e6 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 666.444402] env[70385]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ebef5157-3246-4e31-bc48-bad93d3405e6 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=70385) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 668.445620] env[70385]: CRITICAL neutron [None req-ebef5157-3246-4e31-bc48-bad93d3405e6 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 668.445620] env[70385]: ERROR neutron Traceback (most recent call last): [ 668.445620] env[70385]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 668.445620] env[70385]: ERROR neutron sys.exit(main()) [ 668.445620] env[70385]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 668.445620] env[70385]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 668.445620] env[70385]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 668.445620] env[70385]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 668.445620] env[70385]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 668.445620] env[70385]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 668.445620] env[70385]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 668.445620] env[70385]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 668.445620] env[70385]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 668.445620] env[70385]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 668.445620] env[70385]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 668.445620] env[70385]: ERROR neutron response = self.get(path=path, params=params) [ 668.446291] env[70385]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 668.446291] env[70385]: ERROR neutron raise RuntimeError(msg, last_err) [ 668.446291] env[70385]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 668.446291] env[70385]: ERROR neutron [ 671.055707] env[70574]: INFO neutron.common.config [-] Logging enabled! [ 671.056108] env[70574]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 671.056248] env[70574]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=70574) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 671.056474] env[70574]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 671.056570] env[70574]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 671.057405] env[70574]: WARNING oslo_policy.policy [None req-7bf4703e-367b-4474-9d21-c90a58f9b5fb None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 671.057629] env[70574]: WARNING oslo_policy.policy [None req-7bf4703e-367b-4474-9d21-c90a58f9b5fb None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 671.059375] env[70574]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-7bf4703e-367b-4474-9d21-c90a58f9b5fb None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 671.082977] env[70574]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7bf4703e-367b-4474-9d21-c90a58f9b5fb None None] Session token - acquiring [ 671.083825] env[70574]: DEBUG networking_nsxv3.common.locking [None req-7bf4703e-367b-4474-9d21-c90a58f9b5fb None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70574) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 671.084044] env[70574]: DEBUG oslo_concurrency.lockutils [None req-7bf4703e-367b-4474-9d21-c90a58f9b5fb None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70574) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 671.084177] env[70574]: DEBUG oslo_concurrency.lockutils [None req-7bf4703e-367b-4474-9d21-c90a58f9b5fb None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70574) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 671.084551] env[70574]: DEBUG oslo_concurrency.lockutils [None req-7bf4703e-367b-4474-9d21-c90a58f9b5fb None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70574) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 671.094208] env[70574]: DEBUG oslo_concurrency.lockutils [None req-7bf4703e-367b-4474-9d21-c90a58f9b5fb None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70574) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 671.094443] env[70574]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7bf4703e-367b-4474-9d21-c90a58f9b5fb None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 671.094594] env[70574]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7bf4703e-367b-4474-9d21-c90a58f9b5fb None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=70574) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 673.100676] env[70574]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7bf4703e-367b-4474-9d21-c90a58f9b5fb None None] Session token - acquiring [ 673.101059] env[70574]: DEBUG networking_nsxv3.common.locking [None req-7bf4703e-367b-4474-9d21-c90a58f9b5fb None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70574) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 673.101189] env[70574]: DEBUG oslo_concurrency.lockutils [None req-7bf4703e-367b-4474-9d21-c90a58f9b5fb None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70574) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 673.101313] env[70574]: DEBUG oslo_concurrency.lockutils [None req-7bf4703e-367b-4474-9d21-c90a58f9b5fb None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70574) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 673.101590] env[70574]: DEBUG oslo_concurrency.lockutils [None req-7bf4703e-367b-4474-9d21-c90a58f9b5fb None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70574) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 673.111127] env[70574]: DEBUG oslo_concurrency.lockutils [None req-7bf4703e-367b-4474-9d21-c90a58f9b5fb None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70574) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 673.111370] env[70574]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7bf4703e-367b-4474-9d21-c90a58f9b5fb None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 673.111510] env[70574]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7bf4703e-367b-4474-9d21-c90a58f9b5fb None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=70574) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 675.113457] env[70574]: CRITICAL neutron [None req-7bf4703e-367b-4474-9d21-c90a58f9b5fb None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 675.113457] env[70574]: ERROR neutron Traceback (most recent call last): [ 675.113457] env[70574]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 675.113457] env[70574]: ERROR neutron sys.exit(main()) [ 675.113457] env[70574]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 675.113457] env[70574]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 675.113457] env[70574]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 675.113457] env[70574]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 675.113457] env[70574]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 675.113457] env[70574]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 675.113457] env[70574]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 675.113457] env[70574]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 675.113457] env[70574]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 675.113457] env[70574]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 675.113457] env[70574]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 675.113457] env[70574]: ERROR neutron response = self.get(path=path, params=params) [ 675.114169] env[70574]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 675.114169] env[70574]: ERROR neutron raise RuntimeError(msg, last_err) [ 675.114169] env[70574]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 675.114169] env[70574]: ERROR neutron [ 677.874695] env[70755]: INFO neutron.common.config [-] Logging enabled! [ 677.875082] env[70755]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 677.875245] env[70755]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=70755) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 677.875465] env[70755]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 677.875554] env[70755]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 677.876435] env[70755]: WARNING oslo_policy.policy [None req-4f2c6cf2-2760-499f-ba5a-910f5df90d15 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 677.876645] env[70755]: WARNING oslo_policy.policy [None req-4f2c6cf2-2760-499f-ba5a-910f5df90d15 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 677.877862] env[70755]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-4f2c6cf2-2760-499f-ba5a-910f5df90d15 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 677.914207] env[70755]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4f2c6cf2-2760-499f-ba5a-910f5df90d15 None None] Session token - acquiring [ 677.915032] env[70755]: DEBUG networking_nsxv3.common.locking [None req-4f2c6cf2-2760-499f-ba5a-910f5df90d15 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70755) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 677.915295] env[70755]: DEBUG oslo_concurrency.lockutils [None req-4f2c6cf2-2760-499f-ba5a-910f5df90d15 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70755) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 677.915391] env[70755]: DEBUG oslo_concurrency.lockutils [None req-4f2c6cf2-2760-499f-ba5a-910f5df90d15 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70755) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 677.915735] env[70755]: DEBUG oslo_concurrency.lockutils [None req-4f2c6cf2-2760-499f-ba5a-910f5df90d15 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70755) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 677.925999] env[70755]: DEBUG oslo_concurrency.lockutils [None req-4f2c6cf2-2760-499f-ba5a-910f5df90d15 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70755) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 677.926248] env[70755]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4f2c6cf2-2760-499f-ba5a-910f5df90d15 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 677.926402] env[70755]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4f2c6cf2-2760-499f-ba5a-910f5df90d15 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=70755) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 679.932542] env[70755]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4f2c6cf2-2760-499f-ba5a-910f5df90d15 None None] Session token - acquiring [ 679.932922] env[70755]: DEBUG networking_nsxv3.common.locking [None req-4f2c6cf2-2760-499f-ba5a-910f5df90d15 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70755) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 679.933141] env[70755]: DEBUG oslo_concurrency.lockutils [None req-4f2c6cf2-2760-499f-ba5a-910f5df90d15 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70755) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 679.933271] env[70755]: DEBUG oslo_concurrency.lockutils [None req-4f2c6cf2-2760-499f-ba5a-910f5df90d15 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70755) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 679.933540] env[70755]: DEBUG oslo_concurrency.lockutils [None req-4f2c6cf2-2760-499f-ba5a-910f5df90d15 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70755) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 679.943025] env[70755]: DEBUG oslo_concurrency.lockutils [None req-4f2c6cf2-2760-499f-ba5a-910f5df90d15 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70755) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 679.943264] env[70755]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4f2c6cf2-2760-499f-ba5a-910f5df90d15 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 679.943403] env[70755]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4f2c6cf2-2760-499f-ba5a-910f5df90d15 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=70755) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 681.945004] env[70755]: CRITICAL neutron [None req-4f2c6cf2-2760-499f-ba5a-910f5df90d15 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 681.945004] env[70755]: ERROR neutron Traceback (most recent call last): [ 681.945004] env[70755]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 681.945004] env[70755]: ERROR neutron sys.exit(main()) [ 681.945004] env[70755]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 681.945004] env[70755]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 681.945004] env[70755]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 681.945004] env[70755]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 681.945004] env[70755]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 681.945004] env[70755]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 681.945004] env[70755]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 681.945004] env[70755]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 681.945004] env[70755]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 681.945004] env[70755]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 681.945004] env[70755]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 681.945004] env[70755]: ERROR neutron response = self.get(path=path, params=params) [ 681.945814] env[70755]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 681.945814] env[70755]: ERROR neutron raise RuntimeError(msg, last_err) [ 681.945814] env[70755]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 681.945814] env[70755]: ERROR neutron [ 684.649064] env[70890]: INFO neutron.common.config [-] Logging enabled! [ 684.649405] env[70890]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 684.649595] env[70890]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=70890) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 684.649821] env[70890]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 684.649920] env[70890]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 684.650792] env[70890]: WARNING oslo_policy.policy [None req-b546cdef-8609-48d0-9cac-62dffdcfc960 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 684.650994] env[70890]: WARNING oslo_policy.policy [None req-b546cdef-8609-48d0-9cac-62dffdcfc960 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 684.652287] env[70890]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-b546cdef-8609-48d0-9cac-62dffdcfc960 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 684.667507] env[70890]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b546cdef-8609-48d0-9cac-62dffdcfc960 None None] Session token - acquiring [ 684.668399] env[70890]: DEBUG networking_nsxv3.common.locking [None req-b546cdef-8609-48d0-9cac-62dffdcfc960 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70890) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 684.668636] env[70890]: DEBUG oslo_concurrency.lockutils [None req-b546cdef-8609-48d0-9cac-62dffdcfc960 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70890) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 684.668755] env[70890]: DEBUG oslo_concurrency.lockutils [None req-b546cdef-8609-48d0-9cac-62dffdcfc960 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70890) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 684.669122] env[70890]: DEBUG oslo_concurrency.lockutils [None req-b546cdef-8609-48d0-9cac-62dffdcfc960 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70890) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 684.679095] env[70890]: DEBUG oslo_concurrency.lockutils [None req-b546cdef-8609-48d0-9cac-62dffdcfc960 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70890) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 684.679365] env[70890]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b546cdef-8609-48d0-9cac-62dffdcfc960 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 684.679515] env[70890]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b546cdef-8609-48d0-9cac-62dffdcfc960 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=70890) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 686.688298] env[70890]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b546cdef-8609-48d0-9cac-62dffdcfc960 None None] Session token - acquiring [ 686.688298] env[70890]: DEBUG networking_nsxv3.common.locking [None req-b546cdef-8609-48d0-9cac-62dffdcfc960 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70890) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 686.688298] env[70890]: DEBUG oslo_concurrency.lockutils [None req-b546cdef-8609-48d0-9cac-62dffdcfc960 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70890) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 686.688766] env[70890]: DEBUG oslo_concurrency.lockutils [None req-b546cdef-8609-48d0-9cac-62dffdcfc960 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70890) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 686.688766] env[70890]: DEBUG oslo_concurrency.lockutils [None req-b546cdef-8609-48d0-9cac-62dffdcfc960 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70890) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 686.697677] env[70890]: DEBUG oslo_concurrency.lockutils [None req-b546cdef-8609-48d0-9cac-62dffdcfc960 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70890) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 686.697677] env[70890]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b546cdef-8609-48d0-9cac-62dffdcfc960 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 686.697677] env[70890]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b546cdef-8609-48d0-9cac-62dffdcfc960 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=70890) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 688.698906] env[70890]: CRITICAL neutron [None req-b546cdef-8609-48d0-9cac-62dffdcfc960 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 688.698906] env[70890]: ERROR neutron Traceback (most recent call last): [ 688.698906] env[70890]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 688.698906] env[70890]: ERROR neutron sys.exit(main()) [ 688.698906] env[70890]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 688.698906] env[70890]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 688.698906] env[70890]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 688.698906] env[70890]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 688.698906] env[70890]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 688.698906] env[70890]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 688.698906] env[70890]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 688.698906] env[70890]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 688.698906] env[70890]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 688.698906] env[70890]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 688.698906] env[70890]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 688.698906] env[70890]: ERROR neutron response = self.get(path=path, params=params) [ 688.699641] env[70890]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 688.699641] env[70890]: ERROR neutron raise RuntimeError(msg, last_err) [ 688.699641] env[70890]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 688.699641] env[70890]: ERROR neutron [ 691.372561] env[71198]: INFO neutron.common.config [-] Logging enabled! [ 691.372913] env[71198]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 691.373124] env[71198]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=71198) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 691.373369] env[71198]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 691.373466] env[71198]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 691.374363] env[71198]: WARNING oslo_policy.policy [None req-5a002cae-50c1-46e9-ac6f-c2dfe14fc06c None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 691.374579] env[71198]: WARNING oslo_policy.policy [None req-5a002cae-50c1-46e9-ac6f-c2dfe14fc06c None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 691.375947] env[71198]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-5a002cae-50c1-46e9-ac6f-c2dfe14fc06c None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 691.389726] env[71198]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5a002cae-50c1-46e9-ac6f-c2dfe14fc06c None None] Session token - acquiring [ 691.390743] env[71198]: DEBUG networking_nsxv3.common.locking [None req-5a002cae-50c1-46e9-ac6f-c2dfe14fc06c None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71198) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 691.391008] env[71198]: DEBUG oslo_concurrency.lockutils [None req-5a002cae-50c1-46e9-ac6f-c2dfe14fc06c None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71198) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 691.391068] env[71198]: DEBUG oslo_concurrency.lockutils [None req-5a002cae-50c1-46e9-ac6f-c2dfe14fc06c None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71198) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 691.391426] env[71198]: DEBUG oslo_concurrency.lockutils [None req-5a002cae-50c1-46e9-ac6f-c2dfe14fc06c None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71198) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 691.402248] env[71198]: DEBUG oslo_concurrency.lockutils [None req-5a002cae-50c1-46e9-ac6f-c2dfe14fc06c None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71198) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 691.402248] env[71198]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5a002cae-50c1-46e9-ac6f-c2dfe14fc06c None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 691.402248] env[71198]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5a002cae-50c1-46e9-ac6f-c2dfe14fc06c None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=71198) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 693.410018] env[71198]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5a002cae-50c1-46e9-ac6f-c2dfe14fc06c None None] Session token - acquiring [ 693.410407] env[71198]: DEBUG networking_nsxv3.common.locking [None req-5a002cae-50c1-46e9-ac6f-c2dfe14fc06c None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71198) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 693.410590] env[71198]: DEBUG oslo_concurrency.lockutils [None req-5a002cae-50c1-46e9-ac6f-c2dfe14fc06c None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71198) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 693.410726] env[71198]: DEBUG oslo_concurrency.lockutils [None req-5a002cae-50c1-46e9-ac6f-c2dfe14fc06c None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71198) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 693.411013] env[71198]: DEBUG oslo_concurrency.lockutils [None req-5a002cae-50c1-46e9-ac6f-c2dfe14fc06c None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71198) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 693.420189] env[71198]: DEBUG oslo_concurrency.lockutils [None req-5a002cae-50c1-46e9-ac6f-c2dfe14fc06c None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71198) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 693.420433] env[71198]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5a002cae-50c1-46e9-ac6f-c2dfe14fc06c None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 693.420571] env[71198]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5a002cae-50c1-46e9-ac6f-c2dfe14fc06c None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=71198) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 695.423153] env[71198]: CRITICAL neutron [None req-5a002cae-50c1-46e9-ac6f-c2dfe14fc06c None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 695.423153] env[71198]: ERROR neutron Traceback (most recent call last): [ 695.423153] env[71198]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 695.423153] env[71198]: ERROR neutron sys.exit(main()) [ 695.423153] env[71198]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 695.423153] env[71198]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 695.423153] env[71198]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 695.423153] env[71198]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 695.423153] env[71198]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 695.423153] env[71198]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 695.423153] env[71198]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 695.423153] env[71198]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 695.423153] env[71198]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 695.423153] env[71198]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 695.423153] env[71198]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 695.423153] env[71198]: ERROR neutron response = self.get(path=path, params=params) [ 695.423861] env[71198]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 695.423861] env[71198]: ERROR neutron raise RuntimeError(msg, last_err) [ 695.423861] env[71198]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 695.423861] env[71198]: ERROR neutron [ 698.082817] env[71349]: INFO neutron.common.config [-] Logging enabled! [ 698.083140] env[71349]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 698.083336] env[71349]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=71349) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 698.083559] env[71349]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 698.083652] env[71349]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 698.084512] env[71349]: WARNING oslo_policy.policy [None req-c72a8619-3882-4dae-bd36-2ac595e44ec5 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 698.084726] env[71349]: WARNING oslo_policy.policy [None req-c72a8619-3882-4dae-bd36-2ac595e44ec5 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 698.085952] env[71349]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-c72a8619-3882-4dae-bd36-2ac595e44ec5 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 698.101536] env[71349]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c72a8619-3882-4dae-bd36-2ac595e44ec5 None None] Session token - acquiring [ 698.102400] env[71349]: DEBUG networking_nsxv3.common.locking [None req-c72a8619-3882-4dae-bd36-2ac595e44ec5 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71349) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 698.102604] env[71349]: DEBUG oslo_concurrency.lockutils [None req-c72a8619-3882-4dae-bd36-2ac595e44ec5 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71349) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 698.102731] env[71349]: DEBUG oslo_concurrency.lockutils [None req-c72a8619-3882-4dae-bd36-2ac595e44ec5 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71349) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 698.103092] env[71349]: DEBUG oslo_concurrency.lockutils [None req-c72a8619-3882-4dae-bd36-2ac595e44ec5 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71349) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 698.116325] env[71349]: DEBUG oslo_concurrency.lockutils [None req-c72a8619-3882-4dae-bd36-2ac595e44ec5 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71349) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 698.116594] env[71349]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c72a8619-3882-4dae-bd36-2ac595e44ec5 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 698.116744] env[71349]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c72a8619-3882-4dae-bd36-2ac595e44ec5 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=71349) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 700.122845] env[71349]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c72a8619-3882-4dae-bd36-2ac595e44ec5 None None] Session token - acquiring [ 700.123304] env[71349]: DEBUG networking_nsxv3.common.locking [None req-c72a8619-3882-4dae-bd36-2ac595e44ec5 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71349) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 700.123509] env[71349]: DEBUG oslo_concurrency.lockutils [None req-c72a8619-3882-4dae-bd36-2ac595e44ec5 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71349) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 700.123636] env[71349]: DEBUG oslo_concurrency.lockutils [None req-c72a8619-3882-4dae-bd36-2ac595e44ec5 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71349) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 700.123908] env[71349]: DEBUG oslo_concurrency.lockutils [None req-c72a8619-3882-4dae-bd36-2ac595e44ec5 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71349) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 700.133420] env[71349]: DEBUG oslo_concurrency.lockutils [None req-c72a8619-3882-4dae-bd36-2ac595e44ec5 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71349) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 700.133673] env[71349]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c72a8619-3882-4dae-bd36-2ac595e44ec5 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 700.133812] env[71349]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c72a8619-3882-4dae-bd36-2ac595e44ec5 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=71349) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 702.136789] env[71349]: CRITICAL neutron [None req-c72a8619-3882-4dae-bd36-2ac595e44ec5 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 702.136789] env[71349]: ERROR neutron Traceback (most recent call last): [ 702.136789] env[71349]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 702.136789] env[71349]: ERROR neutron sys.exit(main()) [ 702.136789] env[71349]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 702.136789] env[71349]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 702.136789] env[71349]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 702.136789] env[71349]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 702.136789] env[71349]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 702.136789] env[71349]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 702.136789] env[71349]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 702.136789] env[71349]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 702.136789] env[71349]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 702.136789] env[71349]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 702.136789] env[71349]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 702.136789] env[71349]: ERROR neutron response = self.get(path=path, params=params) [ 702.137577] env[71349]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 702.137577] env[71349]: ERROR neutron raise RuntimeError(msg, last_err) [ 702.137577] env[71349]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 702.137577] env[71349]: ERROR neutron [ 704.843017] env[71472]: INFO neutron.common.config [-] Logging enabled! [ 704.843377] env[71472]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 704.843603] env[71472]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=71472) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 704.844208] env[71472]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 704.844320] env[71472]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 704.845267] env[71472]: WARNING oslo_policy.policy [None req-ae4b3cdc-7223-4f8f-808d-5249d68a932e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 704.845491] env[71472]: WARNING oslo_policy.policy [None req-ae4b3cdc-7223-4f8f-808d-5249d68a932e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 704.853295] env[71472]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-ae4b3cdc-7223-4f8f-808d-5249d68a932e None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 704.868904] env[71472]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ae4b3cdc-7223-4f8f-808d-5249d68a932e None None] Session token - acquiring [ 704.869829] env[71472]: DEBUG networking_nsxv3.common.locking [None req-ae4b3cdc-7223-4f8f-808d-5249d68a932e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71472) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 704.870057] env[71472]: DEBUG oslo_concurrency.lockutils [None req-ae4b3cdc-7223-4f8f-808d-5249d68a932e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71472) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 704.870189] env[71472]: DEBUG oslo_concurrency.lockutils [None req-ae4b3cdc-7223-4f8f-808d-5249d68a932e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71472) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 704.870586] env[71472]: DEBUG oslo_concurrency.lockutils [None req-ae4b3cdc-7223-4f8f-808d-5249d68a932e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71472) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 704.880945] env[71472]: DEBUG oslo_concurrency.lockutils [None req-ae4b3cdc-7223-4f8f-808d-5249d68a932e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71472) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 704.881229] env[71472]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ae4b3cdc-7223-4f8f-808d-5249d68a932e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 704.881382] env[71472]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ae4b3cdc-7223-4f8f-808d-5249d68a932e None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=71472) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 706.888115] env[71472]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ae4b3cdc-7223-4f8f-808d-5249d68a932e None None] Session token - acquiring [ 706.888115] env[71472]: DEBUG networking_nsxv3.common.locking [None req-ae4b3cdc-7223-4f8f-808d-5249d68a932e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71472) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 706.888500] env[71472]: DEBUG oslo_concurrency.lockutils [None req-ae4b3cdc-7223-4f8f-808d-5249d68a932e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71472) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 706.888500] env[71472]: DEBUG oslo_concurrency.lockutils [None req-ae4b3cdc-7223-4f8f-808d-5249d68a932e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71472) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 706.888728] env[71472]: DEBUG oslo_concurrency.lockutils [None req-ae4b3cdc-7223-4f8f-808d-5249d68a932e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71472) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 706.898073] env[71472]: DEBUG oslo_concurrency.lockutils [None req-ae4b3cdc-7223-4f8f-808d-5249d68a932e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71472) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 706.898335] env[71472]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ae4b3cdc-7223-4f8f-808d-5249d68a932e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 706.898529] env[71472]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ae4b3cdc-7223-4f8f-808d-5249d68a932e None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=71472) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 708.900875] env[71472]: CRITICAL neutron [None req-ae4b3cdc-7223-4f8f-808d-5249d68a932e None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 708.900875] env[71472]: ERROR neutron Traceback (most recent call last): [ 708.900875] env[71472]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 708.900875] env[71472]: ERROR neutron sys.exit(main()) [ 708.900875] env[71472]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 708.900875] env[71472]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 708.900875] env[71472]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 708.900875] env[71472]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 708.900875] env[71472]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 708.900875] env[71472]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 708.900875] env[71472]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 708.900875] env[71472]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 708.900875] env[71472]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 708.900875] env[71472]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 708.900875] env[71472]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 708.900875] env[71472]: ERROR neutron response = self.get(path=path, params=params) [ 708.901584] env[71472]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 708.901584] env[71472]: ERROR neutron raise RuntimeError(msg, last_err) [ 708.901584] env[71472]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 708.901584] env[71472]: ERROR neutron [ 711.486389] env[71587]: INFO neutron.common.config [-] Logging enabled! [ 711.486688] env[71587]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 711.486892] env[71587]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=71587) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 711.487114] env[71587]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 711.487211] env[71587]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 711.488075] env[71587]: WARNING oslo_policy.policy [None req-948ffe6e-4771-4aec-a543-beab9c38aa84 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 711.488309] env[71587]: WARNING oslo_policy.policy [None req-948ffe6e-4771-4aec-a543-beab9c38aa84 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 711.489508] env[71587]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-948ffe6e-4771-4aec-a543-beab9c38aa84 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 711.503978] env[71587]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-948ffe6e-4771-4aec-a543-beab9c38aa84 None None] Session token - acquiring [ 711.505270] env[71587]: DEBUG networking_nsxv3.common.locking [None req-948ffe6e-4771-4aec-a543-beab9c38aa84 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71587) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 711.505270] env[71587]: DEBUG oslo_concurrency.lockutils [None req-948ffe6e-4771-4aec-a543-beab9c38aa84 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71587) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 711.505426] env[71587]: DEBUG oslo_concurrency.lockutils [None req-948ffe6e-4771-4aec-a543-beab9c38aa84 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71587) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 711.505460] env[71587]: DEBUG oslo_concurrency.lockutils [None req-948ffe6e-4771-4aec-a543-beab9c38aa84 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71587) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 711.515724] env[71587]: DEBUG oslo_concurrency.lockutils [None req-948ffe6e-4771-4aec-a543-beab9c38aa84 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71587) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 711.515724] env[71587]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-948ffe6e-4771-4aec-a543-beab9c38aa84 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 711.515724] env[71587]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-948ffe6e-4771-4aec-a543-beab9c38aa84 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=71587) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 713.520844] env[71587]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-948ffe6e-4771-4aec-a543-beab9c38aa84 None None] Session token - acquiring [ 713.521253] env[71587]: DEBUG networking_nsxv3.common.locking [None req-948ffe6e-4771-4aec-a543-beab9c38aa84 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71587) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 713.521449] env[71587]: DEBUG oslo_concurrency.lockutils [None req-948ffe6e-4771-4aec-a543-beab9c38aa84 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71587) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 713.521575] env[71587]: DEBUG oslo_concurrency.lockutils [None req-948ffe6e-4771-4aec-a543-beab9c38aa84 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71587) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 713.521846] env[71587]: DEBUG oslo_concurrency.lockutils [None req-948ffe6e-4771-4aec-a543-beab9c38aa84 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71587) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 713.530802] env[71587]: DEBUG oslo_concurrency.lockutils [None req-948ffe6e-4771-4aec-a543-beab9c38aa84 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71587) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 713.531038] env[71587]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-948ffe6e-4771-4aec-a543-beab9c38aa84 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 713.531177] env[71587]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-948ffe6e-4771-4aec-a543-beab9c38aa84 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=71587) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 715.532933] env[71587]: CRITICAL neutron [None req-948ffe6e-4771-4aec-a543-beab9c38aa84 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 715.532933] env[71587]: ERROR neutron Traceback (most recent call last): [ 715.532933] env[71587]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 715.532933] env[71587]: ERROR neutron sys.exit(main()) [ 715.532933] env[71587]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 715.532933] env[71587]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 715.532933] env[71587]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 715.532933] env[71587]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 715.532933] env[71587]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 715.532933] env[71587]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 715.532933] env[71587]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 715.532933] env[71587]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 715.532933] env[71587]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 715.532933] env[71587]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 715.532933] env[71587]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 715.532933] env[71587]: ERROR neutron response = self.get(path=path, params=params) [ 715.533727] env[71587]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 715.533727] env[71587]: ERROR neutron raise RuntimeError(msg, last_err) [ 715.533727] env[71587]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 715.533727] env[71587]: ERROR neutron [ 718.346562] env[71709]: INFO neutron.common.config [-] Logging enabled! [ 718.346837] env[71709]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 718.347093] env[71709]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=71709) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 718.347408] env[71709]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 718.347408] env[71709]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 718.348289] env[71709]: WARNING oslo_policy.policy [None req-ef157e58-2f76-46f5-be91-366392433672 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 718.348518] env[71709]: WARNING oslo_policy.policy [None req-ef157e58-2f76-46f5-be91-366392433672 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 718.349776] env[71709]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-ef157e58-2f76-46f5-be91-366392433672 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 718.364591] env[71709]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ef157e58-2f76-46f5-be91-366392433672 None None] Session token - acquiring [ 718.365470] env[71709]: DEBUG networking_nsxv3.common.locking [None req-ef157e58-2f76-46f5-be91-366392433672 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71709) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 718.365675] env[71709]: DEBUG oslo_concurrency.lockutils [None req-ef157e58-2f76-46f5-be91-366392433672 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71709) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 718.365826] env[71709]: DEBUG oslo_concurrency.lockutils [None req-ef157e58-2f76-46f5-be91-366392433672 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71709) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 718.366237] env[71709]: DEBUG oslo_concurrency.lockutils [None req-ef157e58-2f76-46f5-be91-366392433672 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71709) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 718.375519] env[71709]: DEBUG oslo_concurrency.lockutils [None req-ef157e58-2f76-46f5-be91-366392433672 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71709) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 718.375782] env[71709]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ef157e58-2f76-46f5-be91-366392433672 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 718.375949] env[71709]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ef157e58-2f76-46f5-be91-366392433672 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=71709) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 720.381860] env[71709]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ef157e58-2f76-46f5-be91-366392433672 None None] Session token - acquiring [ 720.382196] env[71709]: DEBUG networking_nsxv3.common.locking [None req-ef157e58-2f76-46f5-be91-366392433672 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71709) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 720.382405] env[71709]: DEBUG oslo_concurrency.lockutils [None req-ef157e58-2f76-46f5-be91-366392433672 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71709) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 720.382534] env[71709]: DEBUG oslo_concurrency.lockutils [None req-ef157e58-2f76-46f5-be91-366392433672 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71709) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 720.382825] env[71709]: DEBUG oslo_concurrency.lockutils [None req-ef157e58-2f76-46f5-be91-366392433672 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71709) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 720.396818] env[71709]: DEBUG oslo_concurrency.lockutils [None req-ef157e58-2f76-46f5-be91-366392433672 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71709) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 720.397110] env[71709]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ef157e58-2f76-46f5-be91-366392433672 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 720.397278] env[71709]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ef157e58-2f76-46f5-be91-366392433672 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=71709) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 722.400345] env[71709]: CRITICAL neutron [None req-ef157e58-2f76-46f5-be91-366392433672 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 722.400345] env[71709]: ERROR neutron Traceback (most recent call last): [ 722.400345] env[71709]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 722.400345] env[71709]: ERROR neutron sys.exit(main()) [ 722.400345] env[71709]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 722.400345] env[71709]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 722.400345] env[71709]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 722.400345] env[71709]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 722.400345] env[71709]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 722.400345] env[71709]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 722.400345] env[71709]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 722.400345] env[71709]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 722.400345] env[71709]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 722.400345] env[71709]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 722.400345] env[71709]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 722.400345] env[71709]: ERROR neutron response = self.get(path=path, params=params) [ 722.401166] env[71709]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 722.401166] env[71709]: ERROR neutron raise RuntimeError(msg, last_err) [ 722.401166] env[71709]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 722.401166] env[71709]: ERROR neutron [ 725.176181] env[71976]: INFO neutron.common.config [-] Logging enabled! [ 725.176523] env[71976]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 725.176748] env[71976]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=71976) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 725.176955] env[71976]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 725.177050] env[71976]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 725.177893] env[71976]: WARNING oslo_policy.policy [None req-b69a9442-8bde-40e4-9e35-0ce44964900f None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 725.178123] env[71976]: WARNING oslo_policy.policy [None req-b69a9442-8bde-40e4-9e35-0ce44964900f None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 725.179442] env[71976]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-b69a9442-8bde-40e4-9e35-0ce44964900f None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 725.194032] env[71976]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b69a9442-8bde-40e4-9e35-0ce44964900f None None] Session token - acquiring [ 725.194795] env[71976]: DEBUG networking_nsxv3.common.locking [None req-b69a9442-8bde-40e4-9e35-0ce44964900f None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71976) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 725.194972] env[71976]: DEBUG oslo_concurrency.lockutils [None req-b69a9442-8bde-40e4-9e35-0ce44964900f None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71976) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 725.195106] env[71976]: DEBUG oslo_concurrency.lockutils [None req-b69a9442-8bde-40e4-9e35-0ce44964900f None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71976) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 725.195473] env[71976]: DEBUG oslo_concurrency.lockutils [None req-b69a9442-8bde-40e4-9e35-0ce44964900f None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71976) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 725.204587] env[71976]: DEBUG oslo_concurrency.lockutils [None req-b69a9442-8bde-40e4-9e35-0ce44964900f None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71976) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 725.204848] env[71976]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b69a9442-8bde-40e4-9e35-0ce44964900f None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 725.205029] env[71976]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b69a9442-8bde-40e4-9e35-0ce44964900f None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=71976) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 727.211387] env[71976]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b69a9442-8bde-40e4-9e35-0ce44964900f None None] Session token - acquiring [ 727.211729] env[71976]: DEBUG networking_nsxv3.common.locking [None req-b69a9442-8bde-40e4-9e35-0ce44964900f None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71976) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 727.211987] env[71976]: DEBUG oslo_concurrency.lockutils [None req-b69a9442-8bde-40e4-9e35-0ce44964900f None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71976) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 727.212072] env[71976]: DEBUG oslo_concurrency.lockutils [None req-b69a9442-8bde-40e4-9e35-0ce44964900f None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71976) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 727.212359] env[71976]: DEBUG oslo_concurrency.lockutils [None req-b69a9442-8bde-40e4-9e35-0ce44964900f None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71976) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 727.222346] env[71976]: DEBUG oslo_concurrency.lockutils [None req-b69a9442-8bde-40e4-9e35-0ce44964900f None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71976) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 727.222605] env[71976]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b69a9442-8bde-40e4-9e35-0ce44964900f None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 727.222821] env[71976]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b69a9442-8bde-40e4-9e35-0ce44964900f None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=71976) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 729.229797] env[71976]: CRITICAL neutron [None req-b69a9442-8bde-40e4-9e35-0ce44964900f None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 729.229797] env[71976]: ERROR neutron Traceback (most recent call last): [ 729.229797] env[71976]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 729.229797] env[71976]: ERROR neutron sys.exit(main()) [ 729.229797] env[71976]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 729.229797] env[71976]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 729.229797] env[71976]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 729.229797] env[71976]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 729.229797] env[71976]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 729.229797] env[71976]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 729.229797] env[71976]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 729.229797] env[71976]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 729.229797] env[71976]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 729.229797] env[71976]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 729.229797] env[71976]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 729.229797] env[71976]: ERROR neutron response = self.get(path=path, params=params) [ 729.230889] env[71976]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 729.230889] env[71976]: ERROR neutron raise RuntimeError(msg, last_err) [ 729.230889] env[71976]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 729.230889] env[71976]: ERROR neutron [ 731.820484] env[72169]: INFO neutron.common.config [-] Logging enabled! [ 731.820841] env[72169]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 731.821000] env[72169]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=72169) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 731.821241] env[72169]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 731.821331] env[72169]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 731.822194] env[72169]: WARNING oslo_policy.policy [None req-7baf2fd6-c9d6-4f2d-842a-2577c1ce13c2 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 731.822393] env[72169]: WARNING oslo_policy.policy [None req-7baf2fd6-c9d6-4f2d-842a-2577c1ce13c2 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 731.823648] env[72169]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-7baf2fd6-c9d6-4f2d-842a-2577c1ce13c2 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 731.837713] env[72169]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7baf2fd6-c9d6-4f2d-842a-2577c1ce13c2 None None] Session token - acquiring [ 731.838587] env[72169]: DEBUG networking_nsxv3.common.locking [None req-7baf2fd6-c9d6-4f2d-842a-2577c1ce13c2 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72169) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 731.838795] env[72169]: DEBUG oslo_concurrency.lockutils [None req-7baf2fd6-c9d6-4f2d-842a-2577c1ce13c2 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72169) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 731.838923] env[72169]: DEBUG oslo_concurrency.lockutils [None req-7baf2fd6-c9d6-4f2d-842a-2577c1ce13c2 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72169) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 731.839288] env[72169]: DEBUG oslo_concurrency.lockutils [None req-7baf2fd6-c9d6-4f2d-842a-2577c1ce13c2 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72169) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 731.848445] env[72169]: DEBUG oslo_concurrency.lockutils [None req-7baf2fd6-c9d6-4f2d-842a-2577c1ce13c2 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72169) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 731.848694] env[72169]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7baf2fd6-c9d6-4f2d-842a-2577c1ce13c2 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 731.848841] env[72169]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7baf2fd6-c9d6-4f2d-842a-2577c1ce13c2 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=72169) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 733.854966] env[72169]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7baf2fd6-c9d6-4f2d-842a-2577c1ce13c2 None None] Session token - acquiring [ 733.855449] env[72169]: DEBUG networking_nsxv3.common.locking [None req-7baf2fd6-c9d6-4f2d-842a-2577c1ce13c2 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72169) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 733.855725] env[72169]: DEBUG oslo_concurrency.lockutils [None req-7baf2fd6-c9d6-4f2d-842a-2577c1ce13c2 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72169) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 733.855910] env[72169]: DEBUG oslo_concurrency.lockutils [None req-7baf2fd6-c9d6-4f2d-842a-2577c1ce13c2 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72169) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 733.856222] env[72169]: DEBUG oslo_concurrency.lockutils [None req-7baf2fd6-c9d6-4f2d-842a-2577c1ce13c2 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72169) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 733.865810] env[72169]: DEBUG oslo_concurrency.lockutils [None req-7baf2fd6-c9d6-4f2d-842a-2577c1ce13c2 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72169) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 733.865992] env[72169]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7baf2fd6-c9d6-4f2d-842a-2577c1ce13c2 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 733.866131] env[72169]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7baf2fd6-c9d6-4f2d-842a-2577c1ce13c2 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=72169) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 735.868885] env[72169]: CRITICAL neutron [None req-7baf2fd6-c9d6-4f2d-842a-2577c1ce13c2 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 735.868885] env[72169]: ERROR neutron Traceback (most recent call last): [ 735.868885] env[72169]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 735.868885] env[72169]: ERROR neutron sys.exit(main()) [ 735.868885] env[72169]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 735.868885] env[72169]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 735.868885] env[72169]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 735.868885] env[72169]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 735.868885] env[72169]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 735.868885] env[72169]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 735.868885] env[72169]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 735.868885] env[72169]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 735.868885] env[72169]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 735.868885] env[72169]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 735.868885] env[72169]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 735.868885] env[72169]: ERROR neutron response = self.get(path=path, params=params) [ 735.869592] env[72169]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 735.869592] env[72169]: ERROR neutron raise RuntimeError(msg, last_err) [ 735.869592] env[72169]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 735.869592] env[72169]: ERROR neutron [ 738.591240] env[72345]: INFO neutron.common.config [-] Logging enabled! [ 738.591549] env[72345]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 738.591824] env[72345]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=72345) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 738.592128] env[72345]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 738.592234] env[72345]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 738.593119] env[72345]: WARNING oslo_policy.policy [None req-fbbbf0b1-fa81-4848-8a9d-a550163e64ed None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 738.593350] env[72345]: WARNING oslo_policy.policy [None req-fbbbf0b1-fa81-4848-8a9d-a550163e64ed None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 738.594629] env[72345]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-fbbbf0b1-fa81-4848-8a9d-a550163e64ed None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 738.623511] env[72345]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fbbbf0b1-fa81-4848-8a9d-a550163e64ed None None] Session token - acquiring [ 738.623511] env[72345]: DEBUG networking_nsxv3.common.locking [None req-fbbbf0b1-fa81-4848-8a9d-a550163e64ed None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72345) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 738.623511] env[72345]: DEBUG oslo_concurrency.lockutils [None req-fbbbf0b1-fa81-4848-8a9d-a550163e64ed None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72345) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 738.623826] env[72345]: DEBUG oslo_concurrency.lockutils [None req-fbbbf0b1-fa81-4848-8a9d-a550163e64ed None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72345) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 738.623826] env[72345]: DEBUG oslo_concurrency.lockutils [None req-fbbbf0b1-fa81-4848-8a9d-a550163e64ed None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72345) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 738.632328] env[72345]: DEBUG oslo_concurrency.lockutils [None req-fbbbf0b1-fa81-4848-8a9d-a550163e64ed None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72345) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 738.632634] env[72345]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fbbbf0b1-fa81-4848-8a9d-a550163e64ed None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 738.632810] env[72345]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fbbbf0b1-fa81-4848-8a9d-a550163e64ed None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=72345) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 740.641056] env[72345]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fbbbf0b1-fa81-4848-8a9d-a550163e64ed None None] Session token - acquiring [ 740.641759] env[72345]: DEBUG networking_nsxv3.common.locking [None req-fbbbf0b1-fa81-4848-8a9d-a550163e64ed None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72345) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 740.642140] env[72345]: DEBUG oslo_concurrency.lockutils [None req-fbbbf0b1-fa81-4848-8a9d-a550163e64ed None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72345) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 740.642413] env[72345]: DEBUG oslo_concurrency.lockutils [None req-fbbbf0b1-fa81-4848-8a9d-a550163e64ed None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72345) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 740.642812] env[72345]: DEBUG oslo_concurrency.lockutils [None req-fbbbf0b1-fa81-4848-8a9d-a550163e64ed None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72345) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 740.652531] env[72345]: DEBUG oslo_concurrency.lockutils [None req-fbbbf0b1-fa81-4848-8a9d-a550163e64ed None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72345) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 740.652865] env[72345]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fbbbf0b1-fa81-4848-8a9d-a550163e64ed None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 740.653114] env[72345]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fbbbf0b1-fa81-4848-8a9d-a550163e64ed None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=72345) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 742.656137] env[72345]: CRITICAL neutron [None req-fbbbf0b1-fa81-4848-8a9d-a550163e64ed None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 742.656137] env[72345]: ERROR neutron Traceback (most recent call last): [ 742.656137] env[72345]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 742.656137] env[72345]: ERROR neutron sys.exit(main()) [ 742.656137] env[72345]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 742.656137] env[72345]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 742.656137] env[72345]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 742.656137] env[72345]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 742.656137] env[72345]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 742.656137] env[72345]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 742.656137] env[72345]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 742.656137] env[72345]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 742.656137] env[72345]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 742.656137] env[72345]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 742.656137] env[72345]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 742.656137] env[72345]: ERROR neutron response = self.get(path=path, params=params) [ 742.657073] env[72345]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 742.657073] env[72345]: ERROR neutron raise RuntimeError(msg, last_err) [ 742.657073] env[72345]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 742.657073] env[72345]: ERROR neutron [ 745.423627] env[72464]: INFO neutron.common.config [-] Logging enabled! [ 745.423929] env[72464]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 745.424239] env[72464]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=72464) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 745.424478] env[72464]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 745.424573] env[72464]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 745.425458] env[72464]: WARNING oslo_policy.policy [None req-7e892d1e-7451-4dbc-9d3c-c3cf119d748a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 745.425678] env[72464]: WARNING oslo_policy.policy [None req-7e892d1e-7451-4dbc-9d3c-c3cf119d748a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 745.426919] env[72464]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-7e892d1e-7451-4dbc-9d3c-c3cf119d748a None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 745.444086] env[72464]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7e892d1e-7451-4dbc-9d3c-c3cf119d748a None None] Session token - acquiring [ 745.454165] env[72464]: DEBUG networking_nsxv3.common.locking [None req-7e892d1e-7451-4dbc-9d3c-c3cf119d748a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72464) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 745.454446] env[72464]: DEBUG oslo_concurrency.lockutils [None req-7e892d1e-7451-4dbc-9d3c-c3cf119d748a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72464) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 745.454578] env[72464]: DEBUG oslo_concurrency.lockutils [None req-7e892d1e-7451-4dbc-9d3c-c3cf119d748a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72464) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 745.455021] env[72464]: DEBUG oslo_concurrency.lockutils [None req-7e892d1e-7451-4dbc-9d3c-c3cf119d748a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72464) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 745.466384] env[72464]: DEBUG oslo_concurrency.lockutils [None req-7e892d1e-7451-4dbc-9d3c-c3cf119d748a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72464) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 745.466639] env[72464]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7e892d1e-7451-4dbc-9d3c-c3cf119d748a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 745.466784] env[72464]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7e892d1e-7451-4dbc-9d3c-c3cf119d748a None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=72464) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 747.472787] env[72464]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7e892d1e-7451-4dbc-9d3c-c3cf119d748a None None] Session token - acquiring [ 747.473164] env[72464]: DEBUG networking_nsxv3.common.locking [None req-7e892d1e-7451-4dbc-9d3c-c3cf119d748a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72464) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 747.473278] env[72464]: DEBUG oslo_concurrency.lockutils [None req-7e892d1e-7451-4dbc-9d3c-c3cf119d748a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72464) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 747.473391] env[72464]: DEBUG oslo_concurrency.lockutils [None req-7e892d1e-7451-4dbc-9d3c-c3cf119d748a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72464) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 747.473666] env[72464]: DEBUG oslo_concurrency.lockutils [None req-7e892d1e-7451-4dbc-9d3c-c3cf119d748a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72464) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 747.482780] env[72464]: DEBUG oslo_concurrency.lockutils [None req-7e892d1e-7451-4dbc-9d3c-c3cf119d748a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72464) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 747.483039] env[72464]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7e892d1e-7451-4dbc-9d3c-c3cf119d748a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 747.483259] env[72464]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7e892d1e-7451-4dbc-9d3c-c3cf119d748a None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=72464) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 749.491213] env[72464]: CRITICAL neutron [None req-7e892d1e-7451-4dbc-9d3c-c3cf119d748a None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 749.491213] env[72464]: ERROR neutron Traceback (most recent call last): [ 749.491213] env[72464]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 749.491213] env[72464]: ERROR neutron sys.exit(main()) [ 749.491213] env[72464]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 749.491213] env[72464]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 749.491213] env[72464]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 749.491213] env[72464]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 749.491213] env[72464]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 749.491213] env[72464]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 749.491213] env[72464]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 749.491213] env[72464]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 749.491213] env[72464]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 749.491213] env[72464]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 749.491213] env[72464]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 749.491213] env[72464]: ERROR neutron response = self.get(path=path, params=params) [ 749.491943] env[72464]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 749.491943] env[72464]: ERROR neutron raise RuntimeError(msg, last_err) [ 749.491943] env[72464]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 749.491943] env[72464]: ERROR neutron [ 752.053590] env[72579]: INFO neutron.common.config [-] Logging enabled! [ 752.053879] env[72579]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 752.054108] env[72579]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=72579) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 752.054341] env[72579]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 752.054427] env[72579]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 752.055339] env[72579]: WARNING oslo_policy.policy [None req-9b2a5a24-18eb-4b9d-b14d-7acb7774d96c None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 752.055515] env[72579]: WARNING oslo_policy.policy [None req-9b2a5a24-18eb-4b9d-b14d-7acb7774d96c None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 752.056757] env[72579]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-9b2a5a24-18eb-4b9d-b14d-7acb7774d96c None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 752.070986] env[72579]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b2a5a24-18eb-4b9d-b14d-7acb7774d96c None None] Session token - acquiring [ 752.071785] env[72579]: DEBUG networking_nsxv3.common.locking [None req-9b2a5a24-18eb-4b9d-b14d-7acb7774d96c None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72579) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 752.071984] env[72579]: DEBUG oslo_concurrency.lockutils [None req-9b2a5a24-18eb-4b9d-b14d-7acb7774d96c None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72579) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 752.072131] env[72579]: DEBUG oslo_concurrency.lockutils [None req-9b2a5a24-18eb-4b9d-b14d-7acb7774d96c None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72579) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 752.072504] env[72579]: DEBUG oslo_concurrency.lockutils [None req-9b2a5a24-18eb-4b9d-b14d-7acb7774d96c None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72579) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 752.081598] env[72579]: DEBUG oslo_concurrency.lockutils [None req-9b2a5a24-18eb-4b9d-b14d-7acb7774d96c None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72579) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 752.081799] env[72579]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b2a5a24-18eb-4b9d-b14d-7acb7774d96c None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 752.081946] env[72579]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b2a5a24-18eb-4b9d-b14d-7acb7774d96c None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=72579) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 754.088126] env[72579]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b2a5a24-18eb-4b9d-b14d-7acb7774d96c None None] Session token - acquiring [ 754.088455] env[72579]: DEBUG networking_nsxv3.common.locking [None req-9b2a5a24-18eb-4b9d-b14d-7acb7774d96c None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72579) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 754.088610] env[72579]: DEBUG oslo_concurrency.lockutils [None req-9b2a5a24-18eb-4b9d-b14d-7acb7774d96c None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72579) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 754.088651] env[72579]: DEBUG oslo_concurrency.lockutils [None req-9b2a5a24-18eb-4b9d-b14d-7acb7774d96c None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72579) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 754.088897] env[72579]: DEBUG oslo_concurrency.lockutils [None req-9b2a5a24-18eb-4b9d-b14d-7acb7774d96c None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72579) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 754.097747] env[72579]: DEBUG oslo_concurrency.lockutils [None req-9b2a5a24-18eb-4b9d-b14d-7acb7774d96c None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72579) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 754.097985] env[72579]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b2a5a24-18eb-4b9d-b14d-7acb7774d96c None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 754.098124] env[72579]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b2a5a24-18eb-4b9d-b14d-7acb7774d96c None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=72579) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 756.100966] env[72579]: CRITICAL neutron [None req-9b2a5a24-18eb-4b9d-b14d-7acb7774d96c None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 756.100966] env[72579]: ERROR neutron Traceback (most recent call last): [ 756.100966] env[72579]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 756.100966] env[72579]: ERROR neutron sys.exit(main()) [ 756.100966] env[72579]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 756.100966] env[72579]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 756.100966] env[72579]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 756.100966] env[72579]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 756.100966] env[72579]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 756.100966] env[72579]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 756.100966] env[72579]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 756.100966] env[72579]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 756.100966] env[72579]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 756.100966] env[72579]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 756.100966] env[72579]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 756.100966] env[72579]: ERROR neutron response = self.get(path=path, params=params) [ 756.101694] env[72579]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 756.101694] env[72579]: ERROR neutron raise RuntimeError(msg, last_err) [ 756.101694] env[72579]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 756.101694] env[72579]: ERROR neutron [ 758.697979] env[72695]: INFO neutron.common.config [-] Logging enabled! [ 758.698328] env[72695]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 758.698577] env[72695]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=72695) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 758.698769] env[72695]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 758.698862] env[72695]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 758.699743] env[72695]: WARNING oslo_policy.policy [None req-53f50512-2a86-4374-9f91-4c8ff7ea4d98 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 758.699960] env[72695]: WARNING oslo_policy.policy [None req-53f50512-2a86-4374-9f91-4c8ff7ea4d98 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 758.701247] env[72695]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-53f50512-2a86-4374-9f91-4c8ff7ea4d98 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 758.717063] env[72695]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-53f50512-2a86-4374-9f91-4c8ff7ea4d98 None None] Session token - acquiring [ 758.718039] env[72695]: DEBUG networking_nsxv3.common.locking [None req-53f50512-2a86-4374-9f91-4c8ff7ea4d98 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72695) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 758.718352] env[72695]: DEBUG oslo_concurrency.lockutils [None req-53f50512-2a86-4374-9f91-4c8ff7ea4d98 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72695) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 758.718552] env[72695]: DEBUG oslo_concurrency.lockutils [None req-53f50512-2a86-4374-9f91-4c8ff7ea4d98 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72695) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 758.718992] env[72695]: DEBUG oslo_concurrency.lockutils [None req-53f50512-2a86-4374-9f91-4c8ff7ea4d98 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72695) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 758.729738] env[72695]: DEBUG oslo_concurrency.lockutils [None req-53f50512-2a86-4374-9f91-4c8ff7ea4d98 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72695) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 758.730085] env[72695]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-53f50512-2a86-4374-9f91-4c8ff7ea4d98 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 758.730302] env[72695]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-53f50512-2a86-4374-9f91-4c8ff7ea4d98 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=72695) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 760.738451] env[72695]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-53f50512-2a86-4374-9f91-4c8ff7ea4d98 None None] Session token - acquiring [ 760.738451] env[72695]: DEBUG networking_nsxv3.common.locking [None req-53f50512-2a86-4374-9f91-4c8ff7ea4d98 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72695) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 760.738451] env[72695]: DEBUG oslo_concurrency.lockutils [None req-53f50512-2a86-4374-9f91-4c8ff7ea4d98 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72695) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 760.738809] env[72695]: DEBUG oslo_concurrency.lockutils [None req-53f50512-2a86-4374-9f91-4c8ff7ea4d98 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72695) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 760.738809] env[72695]: DEBUG oslo_concurrency.lockutils [None req-53f50512-2a86-4374-9f91-4c8ff7ea4d98 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72695) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 760.746865] env[72695]: DEBUG oslo_concurrency.lockutils [None req-53f50512-2a86-4374-9f91-4c8ff7ea4d98 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72695) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 760.747165] env[72695]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-53f50512-2a86-4374-9f91-4c8ff7ea4d98 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 760.747366] env[72695]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-53f50512-2a86-4374-9f91-4c8ff7ea4d98 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=72695) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 762.748939] env[72695]: CRITICAL neutron [None req-53f50512-2a86-4374-9f91-4c8ff7ea4d98 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 762.748939] env[72695]: ERROR neutron Traceback (most recent call last): [ 762.748939] env[72695]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 762.748939] env[72695]: ERROR neutron sys.exit(main()) [ 762.748939] env[72695]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 762.748939] env[72695]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 762.748939] env[72695]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 762.748939] env[72695]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 762.748939] env[72695]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 762.748939] env[72695]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 762.748939] env[72695]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 762.748939] env[72695]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 762.748939] env[72695]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 762.748939] env[72695]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 762.748939] env[72695]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 762.748939] env[72695]: ERROR neutron response = self.get(path=path, params=params) [ 762.749696] env[72695]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 762.749696] env[72695]: ERROR neutron raise RuntimeError(msg, last_err) [ 762.749696] env[72695]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 762.749696] env[72695]: ERROR neutron [ 765.688604] env[72887]: INFO neutron.common.config [-] Logging enabled! [ 765.688604] env[72887]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 765.688604] env[72887]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=72887) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 765.688604] env[72887]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 765.688910] env[72887]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 765.689459] env[72887]: WARNING oslo_policy.policy [None req-08df516b-b075-484a-9696-0bea657eb127 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 765.689684] env[72887]: WARNING oslo_policy.policy [None req-08df516b-b075-484a-9696-0bea657eb127 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 765.690883] env[72887]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-08df516b-b075-484a-9696-0bea657eb127 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 765.705284] env[72887]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-08df516b-b075-484a-9696-0bea657eb127 None None] Session token - acquiring [ 765.706086] env[72887]: DEBUG networking_nsxv3.common.locking [None req-08df516b-b075-484a-9696-0bea657eb127 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72887) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 765.706292] env[72887]: DEBUG oslo_concurrency.lockutils [None req-08df516b-b075-484a-9696-0bea657eb127 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72887) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 765.706417] env[72887]: DEBUG oslo_concurrency.lockutils [None req-08df516b-b075-484a-9696-0bea657eb127 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72887) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 765.706776] env[72887]: DEBUG oslo_concurrency.lockutils [None req-08df516b-b075-484a-9696-0bea657eb127 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72887) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 765.716632] env[72887]: DEBUG oslo_concurrency.lockutils [None req-08df516b-b075-484a-9696-0bea657eb127 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72887) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 765.716890] env[72887]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-08df516b-b075-484a-9696-0bea657eb127 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 765.717129] env[72887]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-08df516b-b075-484a-9696-0bea657eb127 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=72887) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 767.725518] env[72887]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-08df516b-b075-484a-9696-0bea657eb127 None None] Session token - acquiring [ 767.725998] env[72887]: DEBUG networking_nsxv3.common.locking [None req-08df516b-b075-484a-9696-0bea657eb127 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72887) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 767.726268] env[72887]: DEBUG oslo_concurrency.lockutils [None req-08df516b-b075-484a-9696-0bea657eb127 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72887) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 767.726419] env[72887]: DEBUG oslo_concurrency.lockutils [None req-08df516b-b075-484a-9696-0bea657eb127 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72887) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 767.726742] env[72887]: DEBUG oslo_concurrency.lockutils [None req-08df516b-b075-484a-9696-0bea657eb127 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72887) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 767.736951] env[72887]: DEBUG oslo_concurrency.lockutils [None req-08df516b-b075-484a-9696-0bea657eb127 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72887) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 767.737307] env[72887]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-08df516b-b075-484a-9696-0bea657eb127 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 767.738756] env[72887]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-08df516b-b075-484a-9696-0bea657eb127 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=72887) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 769.740940] env[72887]: CRITICAL neutron [None req-08df516b-b075-484a-9696-0bea657eb127 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 769.740940] env[72887]: ERROR neutron Traceback (most recent call last): [ 769.740940] env[72887]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 769.740940] env[72887]: ERROR neutron sys.exit(main()) [ 769.740940] env[72887]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 769.740940] env[72887]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 769.740940] env[72887]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 769.740940] env[72887]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 769.740940] env[72887]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 769.740940] env[72887]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 769.740940] env[72887]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 769.740940] env[72887]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 769.740940] env[72887]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 769.740940] env[72887]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 769.740940] env[72887]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 769.740940] env[72887]: ERROR neutron response = self.get(path=path, params=params) [ 769.741758] env[72887]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 769.741758] env[72887]: ERROR neutron raise RuntimeError(msg, last_err) [ 769.741758] env[72887]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 769.741758] env[72887]: ERROR neutron [ 772.468713] env[73154]: INFO neutron.common.config [-] Logging enabled! [ 772.468713] env[73154]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 772.468713] env[73154]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73154) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 772.468713] env[73154]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 772.468713] env[73154]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 772.469408] env[73154]: WARNING oslo_policy.policy [None req-eb24d116-de5a-403e-ae24-9a34842fb21e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 772.469628] env[73154]: WARNING oslo_policy.policy [None req-eb24d116-de5a-403e-ae24-9a34842fb21e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 772.470858] env[73154]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-eb24d116-de5a-403e-ae24-9a34842fb21e None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 772.486236] env[73154]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-eb24d116-de5a-403e-ae24-9a34842fb21e None None] Session token - acquiring [ 772.487061] env[73154]: DEBUG networking_nsxv3.common.locking [None req-eb24d116-de5a-403e-ae24-9a34842fb21e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73154) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 772.487302] env[73154]: DEBUG oslo_concurrency.lockutils [None req-eb24d116-de5a-403e-ae24-9a34842fb21e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73154) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 772.487423] env[73154]: DEBUG oslo_concurrency.lockutils [None req-eb24d116-de5a-403e-ae24-9a34842fb21e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73154) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 772.487792] env[73154]: DEBUG oslo_concurrency.lockutils [None req-eb24d116-de5a-403e-ae24-9a34842fb21e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73154) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 772.496986] env[73154]: DEBUG oslo_concurrency.lockutils [None req-eb24d116-de5a-403e-ae24-9a34842fb21e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73154) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 772.497278] env[73154]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-eb24d116-de5a-403e-ae24-9a34842fb21e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 772.497391] env[73154]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-eb24d116-de5a-403e-ae24-9a34842fb21e None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73154) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 774.507101] env[73154]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-eb24d116-de5a-403e-ae24-9a34842fb21e None None] Session token - acquiring [ 774.507535] env[73154]: DEBUG networking_nsxv3.common.locking [None req-eb24d116-de5a-403e-ae24-9a34842fb21e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73154) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 774.507732] env[73154]: DEBUG oslo_concurrency.lockutils [None req-eb24d116-de5a-403e-ae24-9a34842fb21e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73154) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 774.507856] env[73154]: DEBUG oslo_concurrency.lockutils [None req-eb24d116-de5a-403e-ae24-9a34842fb21e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73154) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 774.508144] env[73154]: DEBUG oslo_concurrency.lockutils [None req-eb24d116-de5a-403e-ae24-9a34842fb21e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73154) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 774.517667] env[73154]: DEBUG oslo_concurrency.lockutils [None req-eb24d116-de5a-403e-ae24-9a34842fb21e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73154) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 774.517963] env[73154]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-eb24d116-de5a-403e-ae24-9a34842fb21e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 774.518092] env[73154]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-eb24d116-de5a-403e-ae24-9a34842fb21e None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73154) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 776.520876] env[73154]: CRITICAL neutron [None req-eb24d116-de5a-403e-ae24-9a34842fb21e None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 776.520876] env[73154]: ERROR neutron Traceback (most recent call last): [ 776.520876] env[73154]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 776.520876] env[73154]: ERROR neutron sys.exit(main()) [ 776.520876] env[73154]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 776.520876] env[73154]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 776.520876] env[73154]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 776.520876] env[73154]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 776.520876] env[73154]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 776.520876] env[73154]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 776.520876] env[73154]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 776.520876] env[73154]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 776.520876] env[73154]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 776.520876] env[73154]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 776.520876] env[73154]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 776.520876] env[73154]: ERROR neutron response = self.get(path=path, params=params) [ 776.521455] env[73154]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 776.521455] env[73154]: ERROR neutron raise RuntimeError(msg, last_err) [ 776.521455] env[73154]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 776.521455] env[73154]: ERROR neutron [ 779.131213] env[73306]: INFO neutron.common.config [-] Logging enabled! [ 779.131510] env[73306]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 779.131772] env[73306]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73306) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 779.132010] env[73306]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 779.132128] env[73306]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 779.133028] env[73306]: WARNING oslo_policy.policy [None req-beb0e640-00be-4f46-83b9-3a2be8bb4cc7 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 779.133255] env[73306]: WARNING oslo_policy.policy [None req-beb0e640-00be-4f46-83b9-3a2be8bb4cc7 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 779.134525] env[73306]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-beb0e640-00be-4f46-83b9-3a2be8bb4cc7 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 779.149157] env[73306]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-beb0e640-00be-4f46-83b9-3a2be8bb4cc7 None None] Session token - acquiring [ 779.150129] env[73306]: DEBUG networking_nsxv3.common.locking [None req-beb0e640-00be-4f46-83b9-3a2be8bb4cc7 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73306) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 779.150372] env[73306]: DEBUG oslo_concurrency.lockutils [None req-beb0e640-00be-4f46-83b9-3a2be8bb4cc7 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73306) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 779.150524] env[73306]: DEBUG oslo_concurrency.lockutils [None req-beb0e640-00be-4f46-83b9-3a2be8bb4cc7 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73306) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 779.151053] env[73306]: DEBUG oslo_concurrency.lockutils [None req-beb0e640-00be-4f46-83b9-3a2be8bb4cc7 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73306) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 779.160735] env[73306]: DEBUG oslo_concurrency.lockutils [None req-beb0e640-00be-4f46-83b9-3a2be8bb4cc7 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73306) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 779.160999] env[73306]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-beb0e640-00be-4f46-83b9-3a2be8bb4cc7 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 779.161150] env[73306]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-beb0e640-00be-4f46-83b9-3a2be8bb4cc7 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73306) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 781.167308] env[73306]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-beb0e640-00be-4f46-83b9-3a2be8bb4cc7 None None] Session token - acquiring [ 781.167697] env[73306]: DEBUG networking_nsxv3.common.locking [None req-beb0e640-00be-4f46-83b9-3a2be8bb4cc7 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73306) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 781.167896] env[73306]: DEBUG oslo_concurrency.lockutils [None req-beb0e640-00be-4f46-83b9-3a2be8bb4cc7 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73306) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 781.168036] env[73306]: DEBUG oslo_concurrency.lockutils [None req-beb0e640-00be-4f46-83b9-3a2be8bb4cc7 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73306) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 781.168362] env[73306]: DEBUG oslo_concurrency.lockutils [None req-beb0e640-00be-4f46-83b9-3a2be8bb4cc7 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73306) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 781.177952] env[73306]: DEBUG oslo_concurrency.lockutils [None req-beb0e640-00be-4f46-83b9-3a2be8bb4cc7 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73306) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 781.178201] env[73306]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-beb0e640-00be-4f46-83b9-3a2be8bb4cc7 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 781.178343] env[73306]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-beb0e640-00be-4f46-83b9-3a2be8bb4cc7 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73306) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 783.181253] env[73306]: CRITICAL neutron [None req-beb0e640-00be-4f46-83b9-3a2be8bb4cc7 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 783.181253] env[73306]: ERROR neutron Traceback (most recent call last): [ 783.181253] env[73306]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 783.181253] env[73306]: ERROR neutron sys.exit(main()) [ 783.181253] env[73306]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 783.181253] env[73306]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 783.181253] env[73306]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 783.181253] env[73306]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 783.181253] env[73306]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 783.181253] env[73306]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 783.181253] env[73306]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 783.181253] env[73306]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 783.181253] env[73306]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 783.181253] env[73306]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 783.181253] env[73306]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 783.181253] env[73306]: ERROR neutron response = self.get(path=path, params=params) [ 783.182007] env[73306]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 783.182007] env[73306]: ERROR neutron raise RuntimeError(msg, last_err) [ 783.182007] env[73306]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 783.182007] env[73306]: ERROR neutron [ 785.782494] env[73399]: INFO neutron.common.config [-] Logging enabled! [ 785.782818] env[73399]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 785.783035] env[73399]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73399) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 785.783286] env[73399]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 785.783423] env[73399]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 785.784373] env[73399]: WARNING oslo_policy.policy [None req-cac2a1ae-576b-42a2-aca9-63efd2d87c9c None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 785.784584] env[73399]: WARNING oslo_policy.policy [None req-cac2a1ae-576b-42a2-aca9-63efd2d87c9c None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 785.785875] env[73399]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-cac2a1ae-576b-42a2-aca9-63efd2d87c9c None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 785.800372] env[73399]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-cac2a1ae-576b-42a2-aca9-63efd2d87c9c None None] Session token - acquiring [ 785.801223] env[73399]: DEBUG networking_nsxv3.common.locking [None req-cac2a1ae-576b-42a2-aca9-63efd2d87c9c None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73399) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 785.801512] env[73399]: DEBUG oslo_concurrency.lockutils [None req-cac2a1ae-576b-42a2-aca9-63efd2d87c9c None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73399) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 785.801697] env[73399]: DEBUG oslo_concurrency.lockutils [None req-cac2a1ae-576b-42a2-aca9-63efd2d87c9c None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73399) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 785.802115] env[73399]: DEBUG oslo_concurrency.lockutils [None req-cac2a1ae-576b-42a2-aca9-63efd2d87c9c None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73399) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 785.811864] env[73399]: DEBUG oslo_concurrency.lockutils [None req-cac2a1ae-576b-42a2-aca9-63efd2d87c9c None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73399) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 785.812185] env[73399]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-cac2a1ae-576b-42a2-aca9-63efd2d87c9c None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 785.812372] env[73399]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-cac2a1ae-576b-42a2-aca9-63efd2d87c9c None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73399) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 787.818518] env[73399]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-cac2a1ae-576b-42a2-aca9-63efd2d87c9c None None] Session token - acquiring [ 787.820032] env[73399]: DEBUG networking_nsxv3.common.locking [None req-cac2a1ae-576b-42a2-aca9-63efd2d87c9c None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73399) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 787.820032] env[73399]: DEBUG oslo_concurrency.lockutils [None req-cac2a1ae-576b-42a2-aca9-63efd2d87c9c None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73399) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 787.820183] env[73399]: DEBUG oslo_concurrency.lockutils [None req-cac2a1ae-576b-42a2-aca9-63efd2d87c9c None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73399) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 787.820183] env[73399]: DEBUG oslo_concurrency.lockutils [None req-cac2a1ae-576b-42a2-aca9-63efd2d87c9c None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73399) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 787.828843] env[73399]: DEBUG oslo_concurrency.lockutils [None req-cac2a1ae-576b-42a2-aca9-63efd2d87c9c None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73399) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 787.829086] env[73399]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-cac2a1ae-576b-42a2-aca9-63efd2d87c9c None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 787.829228] env[73399]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-cac2a1ae-576b-42a2-aca9-63efd2d87c9c None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73399) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 789.832351] env[73399]: CRITICAL neutron [None req-cac2a1ae-576b-42a2-aca9-63efd2d87c9c None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 789.832351] env[73399]: ERROR neutron Traceback (most recent call last): [ 789.832351] env[73399]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 789.832351] env[73399]: ERROR neutron sys.exit(main()) [ 789.832351] env[73399]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 789.832351] env[73399]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 789.832351] env[73399]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 789.832351] env[73399]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 789.832351] env[73399]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 789.832351] env[73399]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 789.832351] env[73399]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 789.832351] env[73399]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 789.832351] env[73399]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 789.832351] env[73399]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 789.832351] env[73399]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 789.832351] env[73399]: ERROR neutron response = self.get(path=path, params=params) [ 789.832965] env[73399]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 789.832965] env[73399]: ERROR neutron raise RuntimeError(msg, last_err) [ 789.832965] env[73399]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 789.832965] env[73399]: ERROR neutron [ 792.266200] env[73406]: INFO neutron.common.config [-] Logging enabled! [ 792.266683] env[73406]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 792.266683] env[73406]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73406) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 792.266914] env[73406]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 792.267009] env[73406]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 792.267834] env[73406]: WARNING oslo_policy.policy [None req-ebf3cea6-8c60-4637-b6e6-4e301526dde0 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 792.268090] env[73406]: WARNING oslo_policy.policy [None req-ebf3cea6-8c60-4637-b6e6-4e301526dde0 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 792.269309] env[73406]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-ebf3cea6-8c60-4637-b6e6-4e301526dde0 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 792.284011] env[73406]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ebf3cea6-8c60-4637-b6e6-4e301526dde0 None None] Session token - acquiring [ 792.284832] env[73406]: DEBUG networking_nsxv3.common.locking [None req-ebf3cea6-8c60-4637-b6e6-4e301526dde0 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73406) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 792.285059] env[73406]: DEBUG oslo_concurrency.lockutils [None req-ebf3cea6-8c60-4637-b6e6-4e301526dde0 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73406) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 792.285197] env[73406]: DEBUG oslo_concurrency.lockutils [None req-ebf3cea6-8c60-4637-b6e6-4e301526dde0 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73406) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 792.285574] env[73406]: DEBUG oslo_concurrency.lockutils [None req-ebf3cea6-8c60-4637-b6e6-4e301526dde0 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73406) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 792.294287] env[73406]: DEBUG oslo_concurrency.lockutils [None req-ebf3cea6-8c60-4637-b6e6-4e301526dde0 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73406) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 792.294534] env[73406]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ebf3cea6-8c60-4637-b6e6-4e301526dde0 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 792.294686] env[73406]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ebf3cea6-8c60-4637-b6e6-4e301526dde0 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73406) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 794.300921] env[73406]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ebf3cea6-8c60-4637-b6e6-4e301526dde0 None None] Session token - acquiring [ 794.301279] env[73406]: DEBUG networking_nsxv3.common.locking [None req-ebf3cea6-8c60-4637-b6e6-4e301526dde0 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73406) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 794.301498] env[73406]: DEBUG oslo_concurrency.lockutils [None req-ebf3cea6-8c60-4637-b6e6-4e301526dde0 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73406) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 794.301633] env[73406]: DEBUG oslo_concurrency.lockutils [None req-ebf3cea6-8c60-4637-b6e6-4e301526dde0 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73406) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 794.301907] env[73406]: DEBUG oslo_concurrency.lockutils [None req-ebf3cea6-8c60-4637-b6e6-4e301526dde0 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73406) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 794.311284] env[73406]: DEBUG oslo_concurrency.lockutils [None req-ebf3cea6-8c60-4637-b6e6-4e301526dde0 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73406) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 794.311673] env[73406]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ebf3cea6-8c60-4637-b6e6-4e301526dde0 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 794.311917] env[73406]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ebf3cea6-8c60-4637-b6e6-4e301526dde0 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73406) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 796.313330] env[73406]: CRITICAL neutron [None req-ebf3cea6-8c60-4637-b6e6-4e301526dde0 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 796.313330] env[73406]: ERROR neutron Traceback (most recent call last): [ 796.313330] env[73406]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 796.313330] env[73406]: ERROR neutron sys.exit(main()) [ 796.313330] env[73406]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 796.313330] env[73406]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 796.313330] env[73406]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 796.313330] env[73406]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 796.313330] env[73406]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 796.313330] env[73406]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 796.313330] env[73406]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 796.313330] env[73406]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 796.313330] env[73406]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 796.313330] env[73406]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 796.313330] env[73406]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 796.313330] env[73406]: ERROR neutron response = self.get(path=path, params=params) [ 796.313990] env[73406]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 796.313990] env[73406]: ERROR neutron raise RuntimeError(msg, last_err) [ 796.313990] env[73406]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 796.313990] env[73406]: ERROR neutron [ 798.830531] env[73499]: INFO neutron.common.config [-] Logging enabled! [ 798.830842] env[73499]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 798.831038] env[73499]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73499) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 798.831256] env[73499]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 798.831344] env[73499]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 798.832210] env[73499]: WARNING oslo_policy.policy [None req-1ea0caba-930e-435e-9582-912421dd6bd1 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 798.832423] env[73499]: WARNING oslo_policy.policy [None req-1ea0caba-930e-435e-9582-912421dd6bd1 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 798.833798] env[73499]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-1ea0caba-930e-435e-9582-912421dd6bd1 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 798.847743] env[73499]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1ea0caba-930e-435e-9582-912421dd6bd1 None None] Session token - acquiring [ 798.848581] env[73499]: DEBUG networking_nsxv3.common.locking [None req-1ea0caba-930e-435e-9582-912421dd6bd1 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73499) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 798.848784] env[73499]: DEBUG oslo_concurrency.lockutils [None req-1ea0caba-930e-435e-9582-912421dd6bd1 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73499) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 798.848912] env[73499]: DEBUG oslo_concurrency.lockutils [None req-1ea0caba-930e-435e-9582-912421dd6bd1 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73499) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 798.849273] env[73499]: DEBUG oslo_concurrency.lockutils [None req-1ea0caba-930e-435e-9582-912421dd6bd1 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73499) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 798.858153] env[73499]: DEBUG oslo_concurrency.lockutils [None req-1ea0caba-930e-435e-9582-912421dd6bd1 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73499) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 798.858405] env[73499]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1ea0caba-930e-435e-9582-912421dd6bd1 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 798.858552] env[73499]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1ea0caba-930e-435e-9582-912421dd6bd1 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73499) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 800.864462] env[73499]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1ea0caba-930e-435e-9582-912421dd6bd1 None None] Session token - acquiring [ 800.864818] env[73499]: DEBUG networking_nsxv3.common.locking [None req-1ea0caba-930e-435e-9582-912421dd6bd1 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73499) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 800.865012] env[73499]: DEBUG oslo_concurrency.lockutils [None req-1ea0caba-930e-435e-9582-912421dd6bd1 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73499) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 800.865151] env[73499]: DEBUG oslo_concurrency.lockutils [None req-1ea0caba-930e-435e-9582-912421dd6bd1 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73499) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 800.865422] env[73499]: DEBUG oslo_concurrency.lockutils [None req-1ea0caba-930e-435e-9582-912421dd6bd1 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73499) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 800.874371] env[73499]: DEBUG oslo_concurrency.lockutils [None req-1ea0caba-930e-435e-9582-912421dd6bd1 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73499) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 800.874841] env[73499]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1ea0caba-930e-435e-9582-912421dd6bd1 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 800.874841] env[73499]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1ea0caba-930e-435e-9582-912421dd6bd1 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73499) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 802.876974] env[73499]: CRITICAL neutron [None req-1ea0caba-930e-435e-9582-912421dd6bd1 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 802.876974] env[73499]: ERROR neutron Traceback (most recent call last): [ 802.876974] env[73499]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 802.876974] env[73499]: ERROR neutron sys.exit(main()) [ 802.876974] env[73499]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 802.876974] env[73499]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 802.876974] env[73499]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 802.876974] env[73499]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 802.876974] env[73499]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 802.876974] env[73499]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 802.876974] env[73499]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 802.876974] env[73499]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 802.876974] env[73499]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 802.876974] env[73499]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 802.876974] env[73499]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 802.876974] env[73499]: ERROR neutron response = self.get(path=path, params=params) [ 802.877638] env[73499]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 802.877638] env[73499]: ERROR neutron raise RuntimeError(msg, last_err) [ 802.877638] env[73499]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 802.877638] env[73499]: ERROR neutron [ 805.667433] env[73563]: INFO neutron.common.config [-] Logging enabled! [ 805.667731] env[73563]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 805.667959] env[73563]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73563) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 805.668198] env[73563]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 805.668287] env[73563]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 805.669125] env[73563]: WARNING oslo_policy.policy [None req-d8894839-40da-40eb-8891-ae5bfd6b25a6 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 805.669418] env[73563]: WARNING oslo_policy.policy [None req-d8894839-40da-40eb-8891-ae5bfd6b25a6 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 805.670680] env[73563]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-d8894839-40da-40eb-8891-ae5bfd6b25a6 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 805.684787] env[73563]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d8894839-40da-40eb-8891-ae5bfd6b25a6 None None] Session token - acquiring [ 805.685702] env[73563]: DEBUG networking_nsxv3.common.locking [None req-d8894839-40da-40eb-8891-ae5bfd6b25a6 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73563) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 805.685902] env[73563]: DEBUG oslo_concurrency.lockutils [None req-d8894839-40da-40eb-8891-ae5bfd6b25a6 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73563) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 805.686035] env[73563]: DEBUG oslo_concurrency.lockutils [None req-d8894839-40da-40eb-8891-ae5bfd6b25a6 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73563) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 805.686430] env[73563]: DEBUG oslo_concurrency.lockutils [None req-d8894839-40da-40eb-8891-ae5bfd6b25a6 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73563) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 805.695997] env[73563]: DEBUG oslo_concurrency.lockutils [None req-d8894839-40da-40eb-8891-ae5bfd6b25a6 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73563) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 805.696302] env[73563]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d8894839-40da-40eb-8891-ae5bfd6b25a6 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 805.696458] env[73563]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d8894839-40da-40eb-8891-ae5bfd6b25a6 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73563) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 807.704145] env[73563]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d8894839-40da-40eb-8891-ae5bfd6b25a6 None None] Session token - acquiring [ 807.704565] env[73563]: DEBUG networking_nsxv3.common.locking [None req-d8894839-40da-40eb-8891-ae5bfd6b25a6 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73563) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 807.704776] env[73563]: DEBUG oslo_concurrency.lockutils [None req-d8894839-40da-40eb-8891-ae5bfd6b25a6 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73563) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 807.704905] env[73563]: DEBUG oslo_concurrency.lockutils [None req-d8894839-40da-40eb-8891-ae5bfd6b25a6 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73563) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 807.705259] env[73563]: DEBUG oslo_concurrency.lockutils [None req-d8894839-40da-40eb-8891-ae5bfd6b25a6 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73563) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 807.718845] env[73563]: DEBUG oslo_concurrency.lockutils [None req-d8894839-40da-40eb-8891-ae5bfd6b25a6 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73563) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 807.719084] env[73563]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d8894839-40da-40eb-8891-ae5bfd6b25a6 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 807.719224] env[73563]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d8894839-40da-40eb-8891-ae5bfd6b25a6 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73563) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 809.720889] env[73563]: CRITICAL neutron [None req-d8894839-40da-40eb-8891-ae5bfd6b25a6 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 809.720889] env[73563]: ERROR neutron Traceback (most recent call last): [ 809.720889] env[73563]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 809.720889] env[73563]: ERROR neutron sys.exit(main()) [ 809.720889] env[73563]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 809.720889] env[73563]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 809.720889] env[73563]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 809.720889] env[73563]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 809.720889] env[73563]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 809.720889] env[73563]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 809.720889] env[73563]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 809.720889] env[73563]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 809.720889] env[73563]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 809.720889] env[73563]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 809.720889] env[73563]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 809.720889] env[73563]: ERROR neutron response = self.get(path=path, params=params) [ 809.721536] env[73563]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 809.721536] env[73563]: ERROR neutron raise RuntimeError(msg, last_err) [ 809.721536] env[73563]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 809.721536] env[73563]: ERROR neutron [ 812.342971] env[73621]: INFO neutron.common.config [-] Logging enabled! [ 812.343245] env[73621]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 812.343484] env[73621]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73621) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 812.343730] env[73621]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 812.343831] env[73621]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 812.344716] env[73621]: WARNING oslo_policy.policy [None req-0de0f4a5-cce9-4054-8c42-5f4080186310 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 812.344916] env[73621]: WARNING oslo_policy.policy [None req-0de0f4a5-cce9-4054-8c42-5f4080186310 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 812.346644] env[73621]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-0de0f4a5-cce9-4054-8c42-5f4080186310 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 812.361061] env[73621]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0de0f4a5-cce9-4054-8c42-5f4080186310 None None] Session token - acquiring [ 812.361876] env[73621]: DEBUG networking_nsxv3.common.locking [None req-0de0f4a5-cce9-4054-8c42-5f4080186310 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73621) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 812.362083] env[73621]: DEBUG oslo_concurrency.lockutils [None req-0de0f4a5-cce9-4054-8c42-5f4080186310 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73621) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 812.362212] env[73621]: DEBUG oslo_concurrency.lockutils [None req-0de0f4a5-cce9-4054-8c42-5f4080186310 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73621) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 812.362579] env[73621]: DEBUG oslo_concurrency.lockutils [None req-0de0f4a5-cce9-4054-8c42-5f4080186310 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73621) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 812.372343] env[73621]: DEBUG oslo_concurrency.lockutils [None req-0de0f4a5-cce9-4054-8c42-5f4080186310 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73621) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 812.372594] env[73621]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0de0f4a5-cce9-4054-8c42-5f4080186310 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 812.372746] env[73621]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0de0f4a5-cce9-4054-8c42-5f4080186310 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73621) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 814.378777] env[73621]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0de0f4a5-cce9-4054-8c42-5f4080186310 None None] Session token - acquiring [ 814.379151] env[73621]: DEBUG networking_nsxv3.common.locking [None req-0de0f4a5-cce9-4054-8c42-5f4080186310 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73621) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 814.379357] env[73621]: DEBUG oslo_concurrency.lockutils [None req-0de0f4a5-cce9-4054-8c42-5f4080186310 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73621) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 814.379481] env[73621]: DEBUG oslo_concurrency.lockutils [None req-0de0f4a5-cce9-4054-8c42-5f4080186310 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73621) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 814.379750] env[73621]: DEBUG oslo_concurrency.lockutils [None req-0de0f4a5-cce9-4054-8c42-5f4080186310 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73621) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 814.388846] env[73621]: DEBUG oslo_concurrency.lockutils [None req-0de0f4a5-cce9-4054-8c42-5f4080186310 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73621) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 814.389081] env[73621]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0de0f4a5-cce9-4054-8c42-5f4080186310 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 814.389221] env[73621]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0de0f4a5-cce9-4054-8c42-5f4080186310 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73621) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 816.396133] env[73621]: CRITICAL neutron [None req-0de0f4a5-cce9-4054-8c42-5f4080186310 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 816.396133] env[73621]: ERROR neutron Traceback (most recent call last): [ 816.396133] env[73621]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 816.396133] env[73621]: ERROR neutron sys.exit(main()) [ 816.396133] env[73621]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 816.396133] env[73621]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 816.396133] env[73621]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 816.396133] env[73621]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 816.396133] env[73621]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 816.396133] env[73621]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 816.396133] env[73621]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 816.396133] env[73621]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 816.396133] env[73621]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 816.396133] env[73621]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 816.396133] env[73621]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 816.396133] env[73621]: ERROR neutron response = self.get(path=path, params=params) [ 816.396770] env[73621]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 816.396770] env[73621]: ERROR neutron raise RuntimeError(msg, last_err) [ 816.396770] env[73621]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 816.396770] env[73621]: ERROR neutron [ 819.160586] env[73798]: INFO neutron.common.config [-] Logging enabled! [ 819.160955] env[73798]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 819.161429] env[73798]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73798) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 819.161775] env[73798]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 819.161928] env[73798]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 819.163212] env[73798]: WARNING oslo_policy.policy [None req-e961c18c-7c38-4c96-9091-de212e0ad4a7 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 819.163560] env[73798]: WARNING oslo_policy.policy [None req-e961c18c-7c38-4c96-9091-de212e0ad4a7 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 819.165485] env[73798]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-e961c18c-7c38-4c96-9091-de212e0ad4a7 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 819.182514] env[73798]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e961c18c-7c38-4c96-9091-de212e0ad4a7 None None] Session token - acquiring [ 819.183380] env[73798]: DEBUG networking_nsxv3.common.locking [None req-e961c18c-7c38-4c96-9091-de212e0ad4a7 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73798) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 819.183587] env[73798]: DEBUG oslo_concurrency.lockutils [None req-e961c18c-7c38-4c96-9091-de212e0ad4a7 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73798) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 819.183715] env[73798]: DEBUG oslo_concurrency.lockutils [None req-e961c18c-7c38-4c96-9091-de212e0ad4a7 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73798) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 819.184155] env[73798]: DEBUG oslo_concurrency.lockutils [None req-e961c18c-7c38-4c96-9091-de212e0ad4a7 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73798) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 819.193240] env[73798]: DEBUG oslo_concurrency.lockutils [None req-e961c18c-7c38-4c96-9091-de212e0ad4a7 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73798) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 819.193503] env[73798]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e961c18c-7c38-4c96-9091-de212e0ad4a7 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 819.193658] env[73798]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e961c18c-7c38-4c96-9091-de212e0ad4a7 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73798) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 821.199552] env[73798]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e961c18c-7c38-4c96-9091-de212e0ad4a7 None None] Session token - acquiring [ 821.200094] env[73798]: DEBUG networking_nsxv3.common.locking [None req-e961c18c-7c38-4c96-9091-de212e0ad4a7 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73798) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 821.200094] env[73798]: DEBUG oslo_concurrency.lockutils [None req-e961c18c-7c38-4c96-9091-de212e0ad4a7 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73798) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 821.200219] env[73798]: DEBUG oslo_concurrency.lockutils [None req-e961c18c-7c38-4c96-9091-de212e0ad4a7 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73798) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 821.200471] env[73798]: DEBUG oslo_concurrency.lockutils [None req-e961c18c-7c38-4c96-9091-de212e0ad4a7 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73798) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 821.210015] env[73798]: DEBUG oslo_concurrency.lockutils [None req-e961c18c-7c38-4c96-9091-de212e0ad4a7 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73798) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 821.210268] env[73798]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e961c18c-7c38-4c96-9091-de212e0ad4a7 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 821.210410] env[73798]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e961c18c-7c38-4c96-9091-de212e0ad4a7 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73798) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 823.212997] env[73798]: CRITICAL neutron [None req-e961c18c-7c38-4c96-9091-de212e0ad4a7 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 823.212997] env[73798]: ERROR neutron Traceback (most recent call last): [ 823.212997] env[73798]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 823.212997] env[73798]: ERROR neutron sys.exit(main()) [ 823.212997] env[73798]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 823.212997] env[73798]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 823.212997] env[73798]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 823.212997] env[73798]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 823.212997] env[73798]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 823.212997] env[73798]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 823.212997] env[73798]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 823.212997] env[73798]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 823.212997] env[73798]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 823.212997] env[73798]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 823.212997] env[73798]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 823.212997] env[73798]: ERROR neutron response = self.get(path=path, params=params) [ 823.213754] env[73798]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 823.213754] env[73798]: ERROR neutron raise RuntimeError(msg, last_err) [ 823.213754] env[73798]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 823.213754] env[73798]: ERROR neutron [ 825.909102] env[73920]: INFO neutron.common.config [-] Logging enabled! [ 825.909464] env[73920]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 825.909638] env[73920]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73920) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 825.909861] env[73920]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 825.909955] env[73920]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 825.911251] env[73920]: WARNING oslo_policy.policy [None req-bd882c42-6e9e-401b-b6e9-477db450fe0e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 825.911481] env[73920]: WARNING oslo_policy.policy [None req-bd882c42-6e9e-401b-b6e9-477db450fe0e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 825.912754] env[73920]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-bd882c42-6e9e-401b-b6e9-477db450fe0e None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 825.932461] env[73920]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bd882c42-6e9e-401b-b6e9-477db450fe0e None None] Session token - acquiring [ 825.933321] env[73920]: DEBUG networking_nsxv3.common.locking [None req-bd882c42-6e9e-401b-b6e9-477db450fe0e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73920) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 825.933532] env[73920]: DEBUG oslo_concurrency.lockutils [None req-bd882c42-6e9e-401b-b6e9-477db450fe0e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73920) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 825.933658] env[73920]: DEBUG oslo_concurrency.lockutils [None req-bd882c42-6e9e-401b-b6e9-477db450fe0e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73920) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 825.934042] env[73920]: DEBUG oslo_concurrency.lockutils [None req-bd882c42-6e9e-401b-b6e9-477db450fe0e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73920) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 825.952143] env[73920]: DEBUG oslo_concurrency.lockutils [None req-bd882c42-6e9e-401b-b6e9-477db450fe0e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73920) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 825.952449] env[73920]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bd882c42-6e9e-401b-b6e9-477db450fe0e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 825.952599] env[73920]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bd882c42-6e9e-401b-b6e9-477db450fe0e None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73920) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 827.960772] env[73920]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bd882c42-6e9e-401b-b6e9-477db450fe0e None None] Session token - acquiring [ 827.961092] env[73920]: DEBUG networking_nsxv3.common.locking [None req-bd882c42-6e9e-401b-b6e9-477db450fe0e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73920) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 827.961236] env[73920]: DEBUG oslo_concurrency.lockutils [None req-bd882c42-6e9e-401b-b6e9-477db450fe0e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73920) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 827.961354] env[73920]: DEBUG oslo_concurrency.lockutils [None req-bd882c42-6e9e-401b-b6e9-477db450fe0e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73920) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 827.961618] env[73920]: DEBUG oslo_concurrency.lockutils [None req-bd882c42-6e9e-401b-b6e9-477db450fe0e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73920) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 827.971658] env[73920]: DEBUG oslo_concurrency.lockutils [None req-bd882c42-6e9e-401b-b6e9-477db450fe0e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73920) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 827.971900] env[73920]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bd882c42-6e9e-401b-b6e9-477db450fe0e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 827.972048] env[73920]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bd882c42-6e9e-401b-b6e9-477db450fe0e None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73920) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 829.974918] env[73920]: CRITICAL neutron [None req-bd882c42-6e9e-401b-b6e9-477db450fe0e None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 829.974918] env[73920]: ERROR neutron Traceback (most recent call last): [ 829.974918] env[73920]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 829.974918] env[73920]: ERROR neutron sys.exit(main()) [ 829.974918] env[73920]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 829.974918] env[73920]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 829.974918] env[73920]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 829.974918] env[73920]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 829.974918] env[73920]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 829.974918] env[73920]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 829.974918] env[73920]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 829.974918] env[73920]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 829.974918] env[73920]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 829.974918] env[73920]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 829.974918] env[73920]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 829.974918] env[73920]: ERROR neutron response = self.get(path=path, params=params) [ 829.975647] env[73920]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 829.975647] env[73920]: ERROR neutron raise RuntimeError(msg, last_err) [ 829.975647] env[73920]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 829.975647] env[73920]: ERROR neutron [ 832.644727] env[74165]: INFO neutron.common.config [-] Logging enabled! [ 832.645105] env[74165]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 832.645332] env[74165]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74165) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 832.645560] env[74165]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 832.645650] env[74165]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 832.646572] env[74165]: WARNING oslo_policy.policy [None req-01df2896-bcae-4b00-92cf-cca683a61f07 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 832.646792] env[74165]: WARNING oslo_policy.policy [None req-01df2896-bcae-4b00-92cf-cca683a61f07 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 832.648036] env[74165]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-01df2896-bcae-4b00-92cf-cca683a61f07 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 832.662207] env[74165]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-01df2896-bcae-4b00-92cf-cca683a61f07 None None] Session token - acquiring [ 832.663101] env[74165]: DEBUG networking_nsxv3.common.locking [None req-01df2896-bcae-4b00-92cf-cca683a61f07 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74165) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 832.663315] env[74165]: DEBUG oslo_concurrency.lockutils [None req-01df2896-bcae-4b00-92cf-cca683a61f07 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74165) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 832.663440] env[74165]: DEBUG oslo_concurrency.lockutils [None req-01df2896-bcae-4b00-92cf-cca683a61f07 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74165) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 832.663810] env[74165]: DEBUG oslo_concurrency.lockutils [None req-01df2896-bcae-4b00-92cf-cca683a61f07 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74165) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 832.673113] env[74165]: DEBUG oslo_concurrency.lockutils [None req-01df2896-bcae-4b00-92cf-cca683a61f07 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74165) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 832.673370] env[74165]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-01df2896-bcae-4b00-92cf-cca683a61f07 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 832.673523] env[74165]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-01df2896-bcae-4b00-92cf-cca683a61f07 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74165) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 834.682117] env[74165]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-01df2896-bcae-4b00-92cf-cca683a61f07 None None] Session token - acquiring [ 834.684478] env[74165]: DEBUG networking_nsxv3.common.locking [None req-01df2896-bcae-4b00-92cf-cca683a61f07 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74165) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 834.684478] env[74165]: DEBUG oslo_concurrency.lockutils [None req-01df2896-bcae-4b00-92cf-cca683a61f07 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74165) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 834.684613] env[74165]: DEBUG oslo_concurrency.lockutils [None req-01df2896-bcae-4b00-92cf-cca683a61f07 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74165) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 834.684613] env[74165]: DEBUG oslo_concurrency.lockutils [None req-01df2896-bcae-4b00-92cf-cca683a61f07 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74165) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 834.696617] env[74165]: DEBUG oslo_concurrency.lockutils [None req-01df2896-bcae-4b00-92cf-cca683a61f07 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74165) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 834.697386] env[74165]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-01df2896-bcae-4b00-92cf-cca683a61f07 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 834.697386] env[74165]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-01df2896-bcae-4b00-92cf-cca683a61f07 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74165) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 836.699939] env[74165]: CRITICAL neutron [None req-01df2896-bcae-4b00-92cf-cca683a61f07 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 836.699939] env[74165]: ERROR neutron Traceback (most recent call last): [ 836.699939] env[74165]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 836.699939] env[74165]: ERROR neutron sys.exit(main()) [ 836.699939] env[74165]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 836.699939] env[74165]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 836.699939] env[74165]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 836.699939] env[74165]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 836.699939] env[74165]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 836.699939] env[74165]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 836.699939] env[74165]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 836.699939] env[74165]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 836.699939] env[74165]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 836.699939] env[74165]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 836.699939] env[74165]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 836.699939] env[74165]: ERROR neutron response = self.get(path=path, params=params) [ 836.700649] env[74165]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 836.700649] env[74165]: ERROR neutron raise RuntimeError(msg, last_err) [ 836.700649] env[74165]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 836.700649] env[74165]: ERROR neutron [ 839.441869] env[74409]: INFO neutron.common.config [-] Logging enabled! [ 839.442107] env[74409]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 839.442392] env[74409]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74409) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 839.442643] env[74409]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 839.442730] env[74409]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 839.443617] env[74409]: WARNING oslo_policy.policy [None req-479709b8-d7ae-440c-9eac-d3f373ada05c None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 839.443828] env[74409]: WARNING oslo_policy.policy [None req-479709b8-d7ae-440c-9eac-d3f373ada05c None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 839.445073] env[74409]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-479709b8-d7ae-440c-9eac-d3f373ada05c None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 839.459247] env[74409]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-479709b8-d7ae-440c-9eac-d3f373ada05c None None] Session token - acquiring [ 839.460449] env[74409]: DEBUG networking_nsxv3.common.locking [None req-479709b8-d7ae-440c-9eac-d3f373ada05c None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74409) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 839.460687] env[74409]: DEBUG oslo_concurrency.lockutils [None req-479709b8-d7ae-440c-9eac-d3f373ada05c None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74409) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 839.460771] env[74409]: DEBUG oslo_concurrency.lockutils [None req-479709b8-d7ae-440c-9eac-d3f373ada05c None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74409) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 839.461132] env[74409]: DEBUG oslo_concurrency.lockutils [None req-479709b8-d7ae-440c-9eac-d3f373ada05c None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74409) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 839.471376] env[74409]: DEBUG oslo_concurrency.lockutils [None req-479709b8-d7ae-440c-9eac-d3f373ada05c None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74409) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 839.471654] env[74409]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-479709b8-d7ae-440c-9eac-d3f373ada05c None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 839.471806] env[74409]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-479709b8-d7ae-440c-9eac-d3f373ada05c None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74409) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 841.478201] env[74409]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-479709b8-d7ae-440c-9eac-d3f373ada05c None None] Session token - acquiring [ 841.478595] env[74409]: DEBUG networking_nsxv3.common.locking [None req-479709b8-d7ae-440c-9eac-d3f373ada05c None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74409) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 841.478753] env[74409]: DEBUG oslo_concurrency.lockutils [None req-479709b8-d7ae-440c-9eac-d3f373ada05c None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74409) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 841.478878] env[74409]: DEBUG oslo_concurrency.lockutils [None req-479709b8-d7ae-440c-9eac-d3f373ada05c None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74409) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 841.479155] env[74409]: DEBUG oslo_concurrency.lockutils [None req-479709b8-d7ae-440c-9eac-d3f373ada05c None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74409) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 841.490816] env[74409]: DEBUG oslo_concurrency.lockutils [None req-479709b8-d7ae-440c-9eac-d3f373ada05c None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74409) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 841.491102] env[74409]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-479709b8-d7ae-440c-9eac-d3f373ada05c None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 841.491281] env[74409]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-479709b8-d7ae-440c-9eac-d3f373ada05c None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74409) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 843.493706] env[74409]: CRITICAL neutron [None req-479709b8-d7ae-440c-9eac-d3f373ada05c None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 843.493706] env[74409]: ERROR neutron Traceback (most recent call last): [ 843.493706] env[74409]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 843.493706] env[74409]: ERROR neutron sys.exit(main()) [ 843.493706] env[74409]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 843.493706] env[74409]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 843.493706] env[74409]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 843.493706] env[74409]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 843.493706] env[74409]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 843.493706] env[74409]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 843.493706] env[74409]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 843.493706] env[74409]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 843.493706] env[74409]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 843.493706] env[74409]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 843.493706] env[74409]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 843.493706] env[74409]: ERROR neutron response = self.get(path=path, params=params) [ 843.494345] env[74409]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 843.494345] env[74409]: ERROR neutron raise RuntimeError(msg, last_err) [ 843.494345] env[74409]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 843.494345] env[74409]: ERROR neutron [ 846.116120] env[74714]: INFO neutron.common.config [-] Logging enabled! [ 846.116410] env[74714]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 846.116677] env[74714]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74714) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 846.116926] env[74714]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 846.117024] env[74714]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 846.117890] env[74714]: WARNING oslo_policy.policy [None req-98fbd704-3e38-448b-b70e-5b937e07ee2a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 846.118073] env[74714]: WARNING oslo_policy.policy [None req-98fbd704-3e38-448b-b70e-5b937e07ee2a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 846.119354] env[74714]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-98fbd704-3e38-448b-b70e-5b937e07ee2a None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 846.134120] env[74714]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-98fbd704-3e38-448b-b70e-5b937e07ee2a None None] Session token - acquiring [ 846.135094] env[74714]: DEBUG networking_nsxv3.common.locking [None req-98fbd704-3e38-448b-b70e-5b937e07ee2a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74714) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 846.135349] env[74714]: DEBUG oslo_concurrency.lockutils [None req-98fbd704-3e38-448b-b70e-5b937e07ee2a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74714) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 846.135533] env[74714]: DEBUG oslo_concurrency.lockutils [None req-98fbd704-3e38-448b-b70e-5b937e07ee2a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74714) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 846.136060] env[74714]: DEBUG oslo_concurrency.lockutils [None req-98fbd704-3e38-448b-b70e-5b937e07ee2a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74714) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 846.145303] env[74714]: DEBUG oslo_concurrency.lockutils [None req-98fbd704-3e38-448b-b70e-5b937e07ee2a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74714) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 846.145546] env[74714]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-98fbd704-3e38-448b-b70e-5b937e07ee2a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 846.145697] env[74714]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-98fbd704-3e38-448b-b70e-5b937e07ee2a None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74714) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 848.151699] env[74714]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-98fbd704-3e38-448b-b70e-5b937e07ee2a None None] Session token - acquiring [ 848.152076] env[74714]: DEBUG networking_nsxv3.common.locking [None req-98fbd704-3e38-448b-b70e-5b937e07ee2a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74714) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 848.152291] env[74714]: DEBUG oslo_concurrency.lockutils [None req-98fbd704-3e38-448b-b70e-5b937e07ee2a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74714) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 848.152415] env[74714]: DEBUG oslo_concurrency.lockutils [None req-98fbd704-3e38-448b-b70e-5b937e07ee2a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74714) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 848.152681] env[74714]: DEBUG oslo_concurrency.lockutils [None req-98fbd704-3e38-448b-b70e-5b937e07ee2a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74714) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 848.161390] env[74714]: DEBUG oslo_concurrency.lockutils [None req-98fbd704-3e38-448b-b70e-5b937e07ee2a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74714) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 848.161623] env[74714]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-98fbd704-3e38-448b-b70e-5b937e07ee2a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 848.161763] env[74714]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-98fbd704-3e38-448b-b70e-5b937e07ee2a None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74714) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 850.164033] env[74714]: CRITICAL neutron [None req-98fbd704-3e38-448b-b70e-5b937e07ee2a None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 850.164033] env[74714]: ERROR neutron Traceback (most recent call last): [ 850.164033] env[74714]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 850.164033] env[74714]: ERROR neutron sys.exit(main()) [ 850.164033] env[74714]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 850.164033] env[74714]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 850.164033] env[74714]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 850.164033] env[74714]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 850.164033] env[74714]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 850.164033] env[74714]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 850.164033] env[74714]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 850.164033] env[74714]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 850.164033] env[74714]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 850.164033] env[74714]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 850.164033] env[74714]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 850.164033] env[74714]: ERROR neutron response = self.get(path=path, params=params) [ 850.164870] env[74714]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 850.164870] env[74714]: ERROR neutron raise RuntimeError(msg, last_err) [ 850.164870] env[74714]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 850.164870] env[74714]: ERROR neutron [ 852.830676] env[74867]: INFO neutron.common.config [-] Logging enabled! [ 852.831042] env[74867]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 852.831157] env[74867]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74867) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 852.831386] env[74867]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 852.831480] env[74867]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 852.832373] env[74867]: WARNING oslo_policy.policy [None req-388eff7b-b630-4584-98ad-d596e6637518 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 852.832593] env[74867]: WARNING oslo_policy.policy [None req-388eff7b-b630-4584-98ad-d596e6637518 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 852.833786] env[74867]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-388eff7b-b630-4584-98ad-d596e6637518 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 852.847675] env[74867]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-388eff7b-b630-4584-98ad-d596e6637518 None None] Session token - acquiring [ 852.848462] env[74867]: DEBUG networking_nsxv3.common.locking [None req-388eff7b-b630-4584-98ad-d596e6637518 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74867) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 852.848665] env[74867]: DEBUG oslo_concurrency.lockutils [None req-388eff7b-b630-4584-98ad-d596e6637518 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74867) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 852.848792] env[74867]: DEBUG oslo_concurrency.lockutils [None req-388eff7b-b630-4584-98ad-d596e6637518 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74867) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 852.849144] env[74867]: DEBUG oslo_concurrency.lockutils [None req-388eff7b-b630-4584-98ad-d596e6637518 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74867) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 852.857941] env[74867]: DEBUG oslo_concurrency.lockutils [None req-388eff7b-b630-4584-98ad-d596e6637518 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74867) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 852.858184] env[74867]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-388eff7b-b630-4584-98ad-d596e6637518 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 852.858331] env[74867]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-388eff7b-b630-4584-98ad-d596e6637518 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74867) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 854.864320] env[74867]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-388eff7b-b630-4584-98ad-d596e6637518 None None] Session token - acquiring [ 854.864658] env[74867]: DEBUG networking_nsxv3.common.locking [None req-388eff7b-b630-4584-98ad-d596e6637518 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74867) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 854.864852] env[74867]: DEBUG oslo_concurrency.lockutils [None req-388eff7b-b630-4584-98ad-d596e6637518 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74867) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 854.865043] env[74867]: DEBUG oslo_concurrency.lockutils [None req-388eff7b-b630-4584-98ad-d596e6637518 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74867) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 854.865336] env[74867]: DEBUG oslo_concurrency.lockutils [None req-388eff7b-b630-4584-98ad-d596e6637518 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74867) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 854.874152] env[74867]: DEBUG oslo_concurrency.lockutils [None req-388eff7b-b630-4584-98ad-d596e6637518 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74867) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 854.874390] env[74867]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-388eff7b-b630-4584-98ad-d596e6637518 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 854.874527] env[74867]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-388eff7b-b630-4584-98ad-d596e6637518 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74867) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 856.876880] env[74867]: CRITICAL neutron [None req-388eff7b-b630-4584-98ad-d596e6637518 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 856.876880] env[74867]: ERROR neutron Traceback (most recent call last): [ 856.876880] env[74867]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 856.876880] env[74867]: ERROR neutron sys.exit(main()) [ 856.876880] env[74867]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 856.876880] env[74867]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 856.876880] env[74867]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 856.876880] env[74867]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 856.876880] env[74867]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 856.876880] env[74867]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 856.876880] env[74867]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 856.876880] env[74867]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 856.876880] env[74867]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 856.876880] env[74867]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 856.876880] env[74867]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 856.876880] env[74867]: ERROR neutron response = self.get(path=path, params=params) [ 856.877581] env[74867]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 856.877581] env[74867]: ERROR neutron raise RuntimeError(msg, last_err) [ 856.877581] env[74867]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 856.877581] env[74867]: ERROR neutron [ 859.317845] env[75020]: INFO neutron.common.config [-] Logging enabled! [ 859.318200] env[75020]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 859.318350] env[75020]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=75020) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 859.318570] env[75020]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 859.318665] env[75020]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 859.319501] env[75020]: WARNING oslo_policy.policy [None req-dde074b2-a63b-4792-b853-66776b5f65d4 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 859.319718] env[75020]: WARNING oslo_policy.policy [None req-dde074b2-a63b-4792-b853-66776b5f65d4 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 859.320983] env[75020]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-dde074b2-a63b-4792-b853-66776b5f65d4 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 859.335335] env[75020]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-dde074b2-a63b-4792-b853-66776b5f65d4 None None] Session token - acquiring [ 859.336154] env[75020]: DEBUG networking_nsxv3.common.locking [None req-dde074b2-a63b-4792-b853-66776b5f65d4 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75020) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 859.336363] env[75020]: DEBUG oslo_concurrency.lockutils [None req-dde074b2-a63b-4792-b853-66776b5f65d4 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75020) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 859.336490] env[75020]: DEBUG oslo_concurrency.lockutils [None req-dde074b2-a63b-4792-b853-66776b5f65d4 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75020) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 859.336856] env[75020]: DEBUG oslo_concurrency.lockutils [None req-dde074b2-a63b-4792-b853-66776b5f65d4 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75020) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 859.345918] env[75020]: DEBUG oslo_concurrency.lockutils [None req-dde074b2-a63b-4792-b853-66776b5f65d4 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75020) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 859.346197] env[75020]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-dde074b2-a63b-4792-b853-66776b5f65d4 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 859.346355] env[75020]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-dde074b2-a63b-4792-b853-66776b5f65d4 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=75020) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 861.352382] env[75020]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-dde074b2-a63b-4792-b853-66776b5f65d4 None None] Session token - acquiring [ 861.352727] env[75020]: DEBUG networking_nsxv3.common.locking [None req-dde074b2-a63b-4792-b853-66776b5f65d4 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75020) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 861.352930] env[75020]: DEBUG oslo_concurrency.lockutils [None req-dde074b2-a63b-4792-b853-66776b5f65d4 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75020) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 861.353056] env[75020]: DEBUG oslo_concurrency.lockutils [None req-dde074b2-a63b-4792-b853-66776b5f65d4 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75020) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 861.353323] env[75020]: DEBUG oslo_concurrency.lockutils [None req-dde074b2-a63b-4792-b853-66776b5f65d4 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75020) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 861.362435] env[75020]: DEBUG oslo_concurrency.lockutils [None req-dde074b2-a63b-4792-b853-66776b5f65d4 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75020) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 861.362679] env[75020]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-dde074b2-a63b-4792-b853-66776b5f65d4 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 861.362815] env[75020]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-dde074b2-a63b-4792-b853-66776b5f65d4 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=75020) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 863.364905] env[75020]: CRITICAL neutron [None req-dde074b2-a63b-4792-b853-66776b5f65d4 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 863.364905] env[75020]: ERROR neutron Traceback (most recent call last): [ 863.364905] env[75020]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 863.364905] env[75020]: ERROR neutron sys.exit(main()) [ 863.364905] env[75020]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 863.364905] env[75020]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 863.364905] env[75020]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 863.364905] env[75020]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 863.364905] env[75020]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 863.364905] env[75020]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 863.364905] env[75020]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 863.364905] env[75020]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 863.364905] env[75020]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 863.364905] env[75020]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 863.364905] env[75020]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 863.364905] env[75020]: ERROR neutron response = self.get(path=path, params=params) [ 863.365603] env[75020]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 863.365603] env[75020]: ERROR neutron raise RuntimeError(msg, last_err) [ 863.365603] env[75020]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 863.365603] env[75020]: ERROR neutron [ 866.134292] env[75223]: INFO neutron.common.config [-] Logging enabled! [ 866.134669] env[75223]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 866.134847] env[75223]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=75223) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 866.135117] env[75223]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 866.135205] env[75223]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 866.136133] env[75223]: WARNING oslo_policy.policy [None req-c2c7508e-e5a1-4e59-b97f-c992eabc0458 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 866.136370] env[75223]: WARNING oslo_policy.policy [None req-c2c7508e-e5a1-4e59-b97f-c992eabc0458 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 866.137595] env[75223]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-c2c7508e-e5a1-4e59-b97f-c992eabc0458 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 866.151876] env[75223]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c2c7508e-e5a1-4e59-b97f-c992eabc0458 None None] Session token - acquiring [ 866.152657] env[75223]: DEBUG networking_nsxv3.common.locking [None req-c2c7508e-e5a1-4e59-b97f-c992eabc0458 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75223) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 866.152889] env[75223]: DEBUG oslo_concurrency.lockutils [None req-c2c7508e-e5a1-4e59-b97f-c992eabc0458 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75223) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 866.153053] env[75223]: DEBUG oslo_concurrency.lockutils [None req-c2c7508e-e5a1-4e59-b97f-c992eabc0458 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75223) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 866.153428] env[75223]: DEBUG oslo_concurrency.lockutils [None req-c2c7508e-e5a1-4e59-b97f-c992eabc0458 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75223) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 866.162257] env[75223]: DEBUG oslo_concurrency.lockutils [None req-c2c7508e-e5a1-4e59-b97f-c992eabc0458 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75223) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 866.162506] env[75223]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c2c7508e-e5a1-4e59-b97f-c992eabc0458 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 866.162656] env[75223]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c2c7508e-e5a1-4e59-b97f-c992eabc0458 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=75223) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 868.169149] env[75223]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c2c7508e-e5a1-4e59-b97f-c992eabc0458 None None] Session token - acquiring [ 868.169514] env[75223]: DEBUG networking_nsxv3.common.locking [None req-c2c7508e-e5a1-4e59-b97f-c992eabc0458 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75223) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 868.169703] env[75223]: DEBUG oslo_concurrency.lockutils [None req-c2c7508e-e5a1-4e59-b97f-c992eabc0458 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75223) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 868.169836] env[75223]: DEBUG oslo_concurrency.lockutils [None req-c2c7508e-e5a1-4e59-b97f-c992eabc0458 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75223) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 868.170112] env[75223]: DEBUG oslo_concurrency.lockutils [None req-c2c7508e-e5a1-4e59-b97f-c992eabc0458 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75223) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 868.179672] env[75223]: DEBUG oslo_concurrency.lockutils [None req-c2c7508e-e5a1-4e59-b97f-c992eabc0458 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75223) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 868.179731] env[75223]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c2c7508e-e5a1-4e59-b97f-c992eabc0458 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 868.179866] env[75223]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c2c7508e-e5a1-4e59-b97f-c992eabc0458 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=75223) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 870.180922] env[75223]: CRITICAL neutron [None req-c2c7508e-e5a1-4e59-b97f-c992eabc0458 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 870.180922] env[75223]: ERROR neutron Traceback (most recent call last): [ 870.180922] env[75223]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 870.180922] env[75223]: ERROR neutron sys.exit(main()) [ 870.180922] env[75223]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 870.180922] env[75223]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 870.180922] env[75223]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 870.180922] env[75223]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 870.180922] env[75223]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 870.180922] env[75223]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 870.180922] env[75223]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 870.180922] env[75223]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 870.180922] env[75223]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 870.180922] env[75223]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 870.180922] env[75223]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 870.180922] env[75223]: ERROR neutron response = self.get(path=path, params=params) [ 870.181534] env[75223]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 870.181534] env[75223]: ERROR neutron raise RuntimeError(msg, last_err) [ 870.181534] env[75223]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 870.181534] env[75223]: ERROR neutron [ 872.947514] env[75431]: INFO neutron.common.config [-] Logging enabled! [ 872.947840] env[75431]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 872.948120] env[75431]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=75431) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 872.948350] env[75431]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 872.948444] env[75431]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 872.949288] env[75431]: WARNING oslo_policy.policy [None req-e083e791-dff3-4c6e-a373-ba7e44d146c2 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 872.949522] env[75431]: WARNING oslo_policy.policy [None req-e083e791-dff3-4c6e-a373-ba7e44d146c2 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 872.950738] env[75431]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-e083e791-dff3-4c6e-a373-ba7e44d146c2 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 872.965593] env[75431]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e083e791-dff3-4c6e-a373-ba7e44d146c2 None None] Session token - acquiring [ 872.966403] env[75431]: DEBUG networking_nsxv3.common.locking [None req-e083e791-dff3-4c6e-a373-ba7e44d146c2 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75431) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 872.966617] env[75431]: DEBUG oslo_concurrency.lockutils [None req-e083e791-dff3-4c6e-a373-ba7e44d146c2 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75431) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 872.966744] env[75431]: DEBUG oslo_concurrency.lockutils [None req-e083e791-dff3-4c6e-a373-ba7e44d146c2 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75431) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 872.967921] env[75431]: DEBUG oslo_concurrency.lockutils [None req-e083e791-dff3-4c6e-a373-ba7e44d146c2 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75431) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 872.976887] env[75431]: DEBUG oslo_concurrency.lockutils [None req-e083e791-dff3-4c6e-a373-ba7e44d146c2 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75431) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 872.977114] env[75431]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e083e791-dff3-4c6e-a373-ba7e44d146c2 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 872.977261] env[75431]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e083e791-dff3-4c6e-a373-ba7e44d146c2 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=75431) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 874.983360] env[75431]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e083e791-dff3-4c6e-a373-ba7e44d146c2 None None] Session token - acquiring [ 874.983700] env[75431]: DEBUG networking_nsxv3.common.locking [None req-e083e791-dff3-4c6e-a373-ba7e44d146c2 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75431) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 874.983895] env[75431]: DEBUG oslo_concurrency.lockutils [None req-e083e791-dff3-4c6e-a373-ba7e44d146c2 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75431) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 874.984038] env[75431]: DEBUG oslo_concurrency.lockutils [None req-e083e791-dff3-4c6e-a373-ba7e44d146c2 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75431) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 874.984323] env[75431]: DEBUG oslo_concurrency.lockutils [None req-e083e791-dff3-4c6e-a373-ba7e44d146c2 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75431) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 874.993513] env[75431]: DEBUG oslo_concurrency.lockutils [None req-e083e791-dff3-4c6e-a373-ba7e44d146c2 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75431) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 874.993755] env[75431]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e083e791-dff3-4c6e-a373-ba7e44d146c2 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 874.993987] env[75431]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e083e791-dff3-4c6e-a373-ba7e44d146c2 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=75431) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 876.995143] env[75431]: CRITICAL neutron [None req-e083e791-dff3-4c6e-a373-ba7e44d146c2 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 876.995143] env[75431]: ERROR neutron Traceback (most recent call last): [ 876.995143] env[75431]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 876.995143] env[75431]: ERROR neutron sys.exit(main()) [ 876.995143] env[75431]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 876.995143] env[75431]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 876.995143] env[75431]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 876.995143] env[75431]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 876.995143] env[75431]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 876.995143] env[75431]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 876.995143] env[75431]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 876.995143] env[75431]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 876.995143] env[75431]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 876.995143] env[75431]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 876.995143] env[75431]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 876.995143] env[75431]: ERROR neutron response = self.get(path=path, params=params) [ 876.995838] env[75431]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 876.995838] env[75431]: ERROR neutron raise RuntimeError(msg, last_err) [ 876.995838] env[75431]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 876.995838] env[75431]: ERROR neutron [ 879.607229] env[75579]: INFO neutron.common.config [-] Logging enabled! [ 879.607602] env[75579]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 879.607850] env[75579]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=75579) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 879.608100] env[75579]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 879.608199] env[75579]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 879.609084] env[75579]: WARNING oslo_policy.policy [None req-00e3acef-34bb-4fcf-8243-5e5063eb813e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 879.609306] env[75579]: WARNING oslo_policy.policy [None req-00e3acef-34bb-4fcf-8243-5e5063eb813e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 879.610582] env[75579]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-00e3acef-34bb-4fcf-8243-5e5063eb813e None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 879.634010] env[75579]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-00e3acef-34bb-4fcf-8243-5e5063eb813e None None] Session token - acquiring [ 879.635004] env[75579]: DEBUG networking_nsxv3.common.locking [None req-00e3acef-34bb-4fcf-8243-5e5063eb813e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75579) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 879.635246] env[75579]: DEBUG oslo_concurrency.lockutils [None req-00e3acef-34bb-4fcf-8243-5e5063eb813e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75579) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 879.635379] env[75579]: DEBUG oslo_concurrency.lockutils [None req-00e3acef-34bb-4fcf-8243-5e5063eb813e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75579) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 879.635778] env[75579]: DEBUG oslo_concurrency.lockutils [None req-00e3acef-34bb-4fcf-8243-5e5063eb813e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75579) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 879.645243] env[75579]: DEBUG oslo_concurrency.lockutils [None req-00e3acef-34bb-4fcf-8243-5e5063eb813e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75579) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 879.645504] env[75579]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-00e3acef-34bb-4fcf-8243-5e5063eb813e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 879.645658] env[75579]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-00e3acef-34bb-4fcf-8243-5e5063eb813e None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=75579) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 881.651565] env[75579]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-00e3acef-34bb-4fcf-8243-5e5063eb813e None None] Session token - acquiring [ 881.651979] env[75579]: DEBUG networking_nsxv3.common.locking [None req-00e3acef-34bb-4fcf-8243-5e5063eb813e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75579) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 881.652228] env[75579]: DEBUG oslo_concurrency.lockutils [None req-00e3acef-34bb-4fcf-8243-5e5063eb813e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75579) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 881.652359] env[75579]: DEBUG oslo_concurrency.lockutils [None req-00e3acef-34bb-4fcf-8243-5e5063eb813e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75579) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 881.652639] env[75579]: DEBUG oslo_concurrency.lockutils [None req-00e3acef-34bb-4fcf-8243-5e5063eb813e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75579) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 881.662350] env[75579]: DEBUG oslo_concurrency.lockutils [None req-00e3acef-34bb-4fcf-8243-5e5063eb813e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75579) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 881.663444] env[75579]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-00e3acef-34bb-4fcf-8243-5e5063eb813e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 881.663444] env[75579]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-00e3acef-34bb-4fcf-8243-5e5063eb813e None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=75579) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 883.664881] env[75579]: CRITICAL neutron [None req-00e3acef-34bb-4fcf-8243-5e5063eb813e None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 883.664881] env[75579]: ERROR neutron Traceback (most recent call last): [ 883.664881] env[75579]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 883.664881] env[75579]: ERROR neutron sys.exit(main()) [ 883.664881] env[75579]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 883.664881] env[75579]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 883.664881] env[75579]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 883.664881] env[75579]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 883.664881] env[75579]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 883.664881] env[75579]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 883.664881] env[75579]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 883.664881] env[75579]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 883.664881] env[75579]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 883.664881] env[75579]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 883.664881] env[75579]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 883.664881] env[75579]: ERROR neutron response = self.get(path=path, params=params) [ 883.665601] env[75579]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 883.665601] env[75579]: ERROR neutron raise RuntimeError(msg, last_err) [ 883.665601] env[75579]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 883.665601] env[75579]: ERROR neutron [ 886.342035] env[75653]: INFO neutron.common.config [-] Logging enabled! [ 886.342460] env[75653]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 886.342616] env[75653]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=75653) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 886.342839] env[75653]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 886.342924] env[75653]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 886.344050] env[75653]: WARNING oslo_policy.policy [None req-168c7ebc-75ca-4d00-80a8-9c784ff5ef24 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 886.344275] env[75653]: WARNING oslo_policy.policy [None req-168c7ebc-75ca-4d00-80a8-9c784ff5ef24 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 886.345530] env[75653]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-168c7ebc-75ca-4d00-80a8-9c784ff5ef24 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 886.359640] env[75653]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-168c7ebc-75ca-4d00-80a8-9c784ff5ef24 None None] Session token - acquiring [ 886.361024] env[75653]: DEBUG networking_nsxv3.common.locking [None req-168c7ebc-75ca-4d00-80a8-9c784ff5ef24 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75653) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 886.361307] env[75653]: DEBUG oslo_concurrency.lockutils [None req-168c7ebc-75ca-4d00-80a8-9c784ff5ef24 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75653) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 886.361547] env[75653]: DEBUG oslo_concurrency.lockutils [None req-168c7ebc-75ca-4d00-80a8-9c784ff5ef24 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75653) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 886.361924] env[75653]: DEBUG oslo_concurrency.lockutils [None req-168c7ebc-75ca-4d00-80a8-9c784ff5ef24 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75653) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 886.372451] env[75653]: DEBUG oslo_concurrency.lockutils [None req-168c7ebc-75ca-4d00-80a8-9c784ff5ef24 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75653) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 886.372675] env[75653]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-168c7ebc-75ca-4d00-80a8-9c784ff5ef24 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 886.372899] env[75653]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-168c7ebc-75ca-4d00-80a8-9c784ff5ef24 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=75653) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 888.380422] env[75653]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-168c7ebc-75ca-4d00-80a8-9c784ff5ef24 None None] Session token - acquiring [ 888.380715] env[75653]: DEBUG networking_nsxv3.common.locking [None req-168c7ebc-75ca-4d00-80a8-9c784ff5ef24 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75653) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 888.380900] env[75653]: DEBUG oslo_concurrency.lockutils [None req-168c7ebc-75ca-4d00-80a8-9c784ff5ef24 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75653) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 888.381027] env[75653]: DEBUG oslo_concurrency.lockutils [None req-168c7ebc-75ca-4d00-80a8-9c784ff5ef24 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75653) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 888.381318] env[75653]: DEBUG oslo_concurrency.lockutils [None req-168c7ebc-75ca-4d00-80a8-9c784ff5ef24 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75653) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 888.390554] env[75653]: DEBUG oslo_concurrency.lockutils [None req-168c7ebc-75ca-4d00-80a8-9c784ff5ef24 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75653) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 888.390805] env[75653]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-168c7ebc-75ca-4d00-80a8-9c784ff5ef24 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 888.390945] env[75653]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-168c7ebc-75ca-4d00-80a8-9c784ff5ef24 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=75653) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 890.392908] env[75653]: CRITICAL neutron [None req-168c7ebc-75ca-4d00-80a8-9c784ff5ef24 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 890.392908] env[75653]: ERROR neutron Traceback (most recent call last): [ 890.392908] env[75653]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 890.392908] env[75653]: ERROR neutron sys.exit(main()) [ 890.392908] env[75653]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 890.392908] env[75653]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 890.392908] env[75653]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 890.392908] env[75653]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 890.392908] env[75653]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 890.392908] env[75653]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 890.392908] env[75653]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 890.392908] env[75653]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 890.392908] env[75653]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 890.392908] env[75653]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 890.392908] env[75653]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 890.392908] env[75653]: ERROR neutron response = self.get(path=path, params=params) [ 890.393602] env[75653]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 890.393602] env[75653]: ERROR neutron raise RuntimeError(msg, last_err) [ 890.393602] env[75653]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 890.393602] env[75653]: ERROR neutron [ 893.038646] env[75877]: INFO neutron.common.config [-] Logging enabled! [ 893.038945] env[75877]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 893.039199] env[75877]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=75877) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 893.039434] env[75877]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 893.039528] env[75877]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 893.040393] env[75877]: WARNING oslo_policy.policy [None req-c2ad190a-8629-4863-8768-885412efdd1a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 893.040622] env[75877]: WARNING oslo_policy.policy [None req-c2ad190a-8629-4863-8768-885412efdd1a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 893.041821] env[75877]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-c2ad190a-8629-4863-8768-885412efdd1a None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 893.056286] env[75877]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c2ad190a-8629-4863-8768-885412efdd1a None None] Session token - acquiring [ 893.057035] env[75877]: DEBUG networking_nsxv3.common.locking [None req-c2ad190a-8629-4863-8768-885412efdd1a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75877) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 893.057251] env[75877]: DEBUG oslo_concurrency.lockutils [None req-c2ad190a-8629-4863-8768-885412efdd1a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75877) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 893.057377] env[75877]: DEBUG oslo_concurrency.lockutils [None req-c2ad190a-8629-4863-8768-885412efdd1a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75877) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 893.057836] env[75877]: DEBUG oslo_concurrency.lockutils [None req-c2ad190a-8629-4863-8768-885412efdd1a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75877) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 893.066795] env[75877]: DEBUG oslo_concurrency.lockutils [None req-c2ad190a-8629-4863-8768-885412efdd1a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75877) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 893.067041] env[75877]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c2ad190a-8629-4863-8768-885412efdd1a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 893.067194] env[75877]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c2ad190a-8629-4863-8768-885412efdd1a None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=75877) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 895.073388] env[75877]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c2ad190a-8629-4863-8768-885412efdd1a None None] Session token - acquiring [ 895.073746] env[75877]: DEBUG networking_nsxv3.common.locking [None req-c2ad190a-8629-4863-8768-885412efdd1a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75877) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 895.073873] env[75877]: DEBUG oslo_concurrency.lockutils [None req-c2ad190a-8629-4863-8768-885412efdd1a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75877) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 895.073997] env[75877]: DEBUG oslo_concurrency.lockutils [None req-c2ad190a-8629-4863-8768-885412efdd1a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75877) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 895.074278] env[75877]: DEBUG oslo_concurrency.lockutils [None req-c2ad190a-8629-4863-8768-885412efdd1a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75877) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 895.083655] env[75877]: DEBUG oslo_concurrency.lockutils [None req-c2ad190a-8629-4863-8768-885412efdd1a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75877) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 895.083884] env[75877]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c2ad190a-8629-4863-8768-885412efdd1a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 895.084038] env[75877]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c2ad190a-8629-4863-8768-885412efdd1a None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=75877) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 897.086933] env[75877]: CRITICAL neutron [None req-c2ad190a-8629-4863-8768-885412efdd1a None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 897.086933] env[75877]: ERROR neutron Traceback (most recent call last): [ 897.086933] env[75877]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 897.086933] env[75877]: ERROR neutron sys.exit(main()) [ 897.086933] env[75877]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 897.086933] env[75877]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 897.086933] env[75877]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 897.086933] env[75877]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 897.086933] env[75877]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 897.086933] env[75877]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 897.086933] env[75877]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 897.086933] env[75877]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 897.086933] env[75877]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 897.086933] env[75877]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 897.086933] env[75877]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 897.086933] env[75877]: ERROR neutron response = self.get(path=path, params=params) [ 897.087656] env[75877]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 897.087656] env[75877]: ERROR neutron raise RuntimeError(msg, last_err) [ 897.087656] env[75877]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 897.087656] env[75877]: ERROR neutron [ 899.574025] env[75923]: INFO neutron.common.config [-] Logging enabled! [ 899.574399] env[75923]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 899.574688] env[75923]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=75923) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 899.574801] env[75923]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 899.574871] env[75923]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 899.575735] env[75923]: WARNING oslo_policy.policy [None req-89b3b489-b1d4-48ed-8264-8ec3da3ebcd2 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 899.575957] env[75923]: WARNING oslo_policy.policy [None req-89b3b489-b1d4-48ed-8264-8ec3da3ebcd2 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 899.577171] env[75923]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-89b3b489-b1d4-48ed-8264-8ec3da3ebcd2 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 899.591335] env[75923]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-89b3b489-b1d4-48ed-8264-8ec3da3ebcd2 None None] Session token - acquiring [ 899.592183] env[75923]: DEBUG networking_nsxv3.common.locking [None req-89b3b489-b1d4-48ed-8264-8ec3da3ebcd2 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75923) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 899.592397] env[75923]: DEBUG oslo_concurrency.lockutils [None req-89b3b489-b1d4-48ed-8264-8ec3da3ebcd2 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75923) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 899.592521] env[75923]: DEBUG oslo_concurrency.lockutils [None req-89b3b489-b1d4-48ed-8264-8ec3da3ebcd2 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75923) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 899.592893] env[75923]: DEBUG oslo_concurrency.lockutils [None req-89b3b489-b1d4-48ed-8264-8ec3da3ebcd2 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75923) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 899.602166] env[75923]: DEBUG oslo_concurrency.lockutils [None req-89b3b489-b1d4-48ed-8264-8ec3da3ebcd2 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75923) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 899.602422] env[75923]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-89b3b489-b1d4-48ed-8264-8ec3da3ebcd2 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 899.602570] env[75923]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-89b3b489-b1d4-48ed-8264-8ec3da3ebcd2 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=75923) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 901.608552] env[75923]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-89b3b489-b1d4-48ed-8264-8ec3da3ebcd2 None None] Session token - acquiring [ 901.608912] env[75923]: DEBUG networking_nsxv3.common.locking [None req-89b3b489-b1d4-48ed-8264-8ec3da3ebcd2 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75923) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 901.609106] env[75923]: DEBUG oslo_concurrency.lockutils [None req-89b3b489-b1d4-48ed-8264-8ec3da3ebcd2 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75923) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 901.609224] env[75923]: DEBUG oslo_concurrency.lockutils [None req-89b3b489-b1d4-48ed-8264-8ec3da3ebcd2 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75923) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 901.609491] env[75923]: DEBUG oslo_concurrency.lockutils [None req-89b3b489-b1d4-48ed-8264-8ec3da3ebcd2 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75923) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 901.618772] env[75923]: DEBUG oslo_concurrency.lockutils [None req-89b3b489-b1d4-48ed-8264-8ec3da3ebcd2 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75923) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 901.619128] env[75923]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-89b3b489-b1d4-48ed-8264-8ec3da3ebcd2 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 901.619326] env[75923]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-89b3b489-b1d4-48ed-8264-8ec3da3ebcd2 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=75923) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 903.620882] env[75923]: CRITICAL neutron [None req-89b3b489-b1d4-48ed-8264-8ec3da3ebcd2 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 903.620882] env[75923]: ERROR neutron Traceback (most recent call last): [ 903.620882] env[75923]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 903.620882] env[75923]: ERROR neutron sys.exit(main()) [ 903.620882] env[75923]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 903.620882] env[75923]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 903.620882] env[75923]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 903.620882] env[75923]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 903.620882] env[75923]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 903.620882] env[75923]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 903.620882] env[75923]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 903.620882] env[75923]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 903.620882] env[75923]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 903.620882] env[75923]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 903.620882] env[75923]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 903.620882] env[75923]: ERROR neutron response = self.get(path=path, params=params) [ 903.621626] env[75923]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 903.621626] env[75923]: ERROR neutron raise RuntimeError(msg, last_err) [ 903.621626] env[75923]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 903.621626] env[75923]: ERROR neutron [ 906.095836] env[75984]: INFO neutron.common.config [-] Logging enabled! [ 906.096286] env[75984]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 906.096592] env[75984]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=75984) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 906.096846] env[75984]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 906.096942] env[75984]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 906.098537] env[75984]: WARNING oslo_policy.policy [None req-3dc65274-37cc-40db-a558-01e9b8c9a9ec None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 906.098537] env[75984]: WARNING oslo_policy.policy [None req-3dc65274-37cc-40db-a558-01e9b8c9a9ec None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 906.099588] env[75984]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-3dc65274-37cc-40db-a558-01e9b8c9a9ec None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 906.115225] env[75984]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3dc65274-37cc-40db-a558-01e9b8c9a9ec None None] Session token - acquiring [ 906.116146] env[75984]: DEBUG networking_nsxv3.common.locking [None req-3dc65274-37cc-40db-a558-01e9b8c9a9ec None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75984) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 906.116383] env[75984]: DEBUG oslo_concurrency.lockutils [None req-3dc65274-37cc-40db-a558-01e9b8c9a9ec None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75984) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 906.116520] env[75984]: DEBUG oslo_concurrency.lockutils [None req-3dc65274-37cc-40db-a558-01e9b8c9a9ec None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75984) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 906.116919] env[75984]: DEBUG oslo_concurrency.lockutils [None req-3dc65274-37cc-40db-a558-01e9b8c9a9ec None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75984) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 906.127152] env[75984]: DEBUG oslo_concurrency.lockutils [None req-3dc65274-37cc-40db-a558-01e9b8c9a9ec None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75984) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 906.127467] env[75984]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3dc65274-37cc-40db-a558-01e9b8c9a9ec None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 906.127624] env[75984]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3dc65274-37cc-40db-a558-01e9b8c9a9ec None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=75984) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 908.133569] env[75984]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3dc65274-37cc-40db-a558-01e9b8c9a9ec None None] Session token - acquiring [ 908.133918] env[75984]: DEBUG networking_nsxv3.common.locking [None req-3dc65274-37cc-40db-a558-01e9b8c9a9ec None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75984) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 908.134206] env[75984]: DEBUG oslo_concurrency.lockutils [None req-3dc65274-37cc-40db-a558-01e9b8c9a9ec None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75984) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 908.134262] env[75984]: DEBUG oslo_concurrency.lockutils [None req-3dc65274-37cc-40db-a558-01e9b8c9a9ec None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75984) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 908.134602] env[75984]: DEBUG oslo_concurrency.lockutils [None req-3dc65274-37cc-40db-a558-01e9b8c9a9ec None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75984) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 908.143383] env[75984]: DEBUG oslo_concurrency.lockutils [None req-3dc65274-37cc-40db-a558-01e9b8c9a9ec None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75984) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 908.143617] env[75984]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3dc65274-37cc-40db-a558-01e9b8c9a9ec None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 908.143756] env[75984]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3dc65274-37cc-40db-a558-01e9b8c9a9ec None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=75984) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 910.144905] env[75984]: CRITICAL neutron [None req-3dc65274-37cc-40db-a558-01e9b8c9a9ec None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 910.144905] env[75984]: ERROR neutron Traceback (most recent call last): [ 910.144905] env[75984]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 910.144905] env[75984]: ERROR neutron sys.exit(main()) [ 910.144905] env[75984]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 910.144905] env[75984]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 910.144905] env[75984]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 910.144905] env[75984]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 910.144905] env[75984]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 910.144905] env[75984]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 910.144905] env[75984]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 910.144905] env[75984]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 910.144905] env[75984]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 910.144905] env[75984]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 910.144905] env[75984]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 910.144905] env[75984]: ERROR neutron response = self.get(path=path, params=params) [ 910.145610] env[75984]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 910.145610] env[75984]: ERROR neutron raise RuntimeError(msg, last_err) [ 910.145610] env[75984]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 910.145610] env[75984]: ERROR neutron [ 912.895993] env[76076]: INFO neutron.common.config [-] Logging enabled! [ 912.896403] env[76076]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 912.896637] env[76076]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76076) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 912.896884] env[76076]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 912.896978] env[76076]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 912.897998] env[76076]: WARNING oslo_policy.policy [None req-79c4313f-45e8-4178-b5b0-9a980400a559 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 912.898228] env[76076]: WARNING oslo_policy.policy [None req-79c4313f-45e8-4178-b5b0-9a980400a559 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 912.899966] env[76076]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-79c4313f-45e8-4178-b5b0-9a980400a559 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 912.915632] env[76076]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-79c4313f-45e8-4178-b5b0-9a980400a559 None None] Session token - acquiring [ 912.917170] env[76076]: DEBUG networking_nsxv3.common.locking [None req-79c4313f-45e8-4178-b5b0-9a980400a559 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76076) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 912.917466] env[76076]: DEBUG oslo_concurrency.lockutils [None req-79c4313f-45e8-4178-b5b0-9a980400a559 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76076) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 912.917639] env[76076]: DEBUG oslo_concurrency.lockutils [None req-79c4313f-45e8-4178-b5b0-9a980400a559 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76076) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 912.918070] env[76076]: DEBUG oslo_concurrency.lockutils [None req-79c4313f-45e8-4178-b5b0-9a980400a559 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76076) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 912.929747] env[76076]: DEBUG oslo_concurrency.lockutils [None req-79c4313f-45e8-4178-b5b0-9a980400a559 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76076) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 912.930346] env[76076]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-79c4313f-45e8-4178-b5b0-9a980400a559 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 912.930615] env[76076]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-79c4313f-45e8-4178-b5b0-9a980400a559 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76076) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 914.936633] env[76076]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-79c4313f-45e8-4178-b5b0-9a980400a559 None None] Session token - acquiring [ 914.936938] env[76076]: DEBUG networking_nsxv3.common.locking [None req-79c4313f-45e8-4178-b5b0-9a980400a559 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76076) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 914.937133] env[76076]: DEBUG oslo_concurrency.lockutils [None req-79c4313f-45e8-4178-b5b0-9a980400a559 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76076) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 914.937287] env[76076]: DEBUG oslo_concurrency.lockutils [None req-79c4313f-45e8-4178-b5b0-9a980400a559 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76076) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 914.937583] env[76076]: DEBUG oslo_concurrency.lockutils [None req-79c4313f-45e8-4178-b5b0-9a980400a559 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76076) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 914.946719] env[76076]: DEBUG oslo_concurrency.lockutils [None req-79c4313f-45e8-4178-b5b0-9a980400a559 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76076) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 914.946959] env[76076]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-79c4313f-45e8-4178-b5b0-9a980400a559 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 914.947129] env[76076]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-79c4313f-45e8-4178-b5b0-9a980400a559 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76076) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 916.948935] env[76076]: CRITICAL neutron [None req-79c4313f-45e8-4178-b5b0-9a980400a559 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 916.948935] env[76076]: ERROR neutron Traceback (most recent call last): [ 916.948935] env[76076]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 916.948935] env[76076]: ERROR neutron sys.exit(main()) [ 916.948935] env[76076]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 916.948935] env[76076]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 916.948935] env[76076]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 916.948935] env[76076]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 916.948935] env[76076]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 916.948935] env[76076]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 916.948935] env[76076]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 916.948935] env[76076]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 916.948935] env[76076]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 916.948935] env[76076]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 916.948935] env[76076]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 916.948935] env[76076]: ERROR neutron response = self.get(path=path, params=params) [ 916.949604] env[76076]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 916.949604] env[76076]: ERROR neutron raise RuntimeError(msg, last_err) [ 916.949604] env[76076]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 916.949604] env[76076]: ERROR neutron [ 919.581761] env[76206]: INFO neutron.common.config [-] Logging enabled! [ 919.582038] env[76206]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 919.582351] env[76206]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76206) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 919.582584] env[76206]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 919.582693] env[76206]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 919.583592] env[76206]: WARNING oslo_policy.policy [None req-0e50c3ce-029d-4328-a882-11fd266cc03f None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 919.583808] env[76206]: WARNING oslo_policy.policy [None req-0e50c3ce-029d-4328-a882-11fd266cc03f None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 919.585076] env[76206]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-0e50c3ce-029d-4328-a882-11fd266cc03f None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 919.609483] env[76206]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0e50c3ce-029d-4328-a882-11fd266cc03f None None] Session token - acquiring [ 919.610427] env[76206]: DEBUG networking_nsxv3.common.locking [None req-0e50c3ce-029d-4328-a882-11fd266cc03f None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76206) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 919.610667] env[76206]: DEBUG oslo_concurrency.lockutils [None req-0e50c3ce-029d-4328-a882-11fd266cc03f None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76206) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 919.610822] env[76206]: DEBUG oslo_concurrency.lockutils [None req-0e50c3ce-029d-4328-a882-11fd266cc03f None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76206) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 919.611241] env[76206]: DEBUG oslo_concurrency.lockutils [None req-0e50c3ce-029d-4328-a882-11fd266cc03f None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76206) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 919.620836] env[76206]: DEBUG oslo_concurrency.lockutils [None req-0e50c3ce-029d-4328-a882-11fd266cc03f None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76206) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 919.621091] env[76206]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0e50c3ce-029d-4328-a882-11fd266cc03f None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 919.621236] env[76206]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0e50c3ce-029d-4328-a882-11fd266cc03f None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76206) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 921.627232] env[76206]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0e50c3ce-029d-4328-a882-11fd266cc03f None None] Session token - acquiring [ 921.627568] env[76206]: DEBUG networking_nsxv3.common.locking [None req-0e50c3ce-029d-4328-a882-11fd266cc03f None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76206) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 921.627798] env[76206]: DEBUG oslo_concurrency.lockutils [None req-0e50c3ce-029d-4328-a882-11fd266cc03f None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76206) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 921.627930] env[76206]: DEBUG oslo_concurrency.lockutils [None req-0e50c3ce-029d-4328-a882-11fd266cc03f None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76206) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 921.628256] env[76206]: DEBUG oslo_concurrency.lockutils [None req-0e50c3ce-029d-4328-a882-11fd266cc03f None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76206) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 921.637169] env[76206]: DEBUG oslo_concurrency.lockutils [None req-0e50c3ce-029d-4328-a882-11fd266cc03f None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76206) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 921.637414] env[76206]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0e50c3ce-029d-4328-a882-11fd266cc03f None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 921.637548] env[76206]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0e50c3ce-029d-4328-a882-11fd266cc03f None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76206) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 923.643998] env[76206]: CRITICAL neutron [None req-0e50c3ce-029d-4328-a882-11fd266cc03f None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 923.643998] env[76206]: ERROR neutron Traceback (most recent call last): [ 923.643998] env[76206]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 923.643998] env[76206]: ERROR neutron sys.exit(main()) [ 923.643998] env[76206]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 923.643998] env[76206]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 923.643998] env[76206]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 923.643998] env[76206]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 923.643998] env[76206]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 923.643998] env[76206]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 923.643998] env[76206]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 923.643998] env[76206]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 923.643998] env[76206]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 923.643998] env[76206]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 923.643998] env[76206]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 923.643998] env[76206]: ERROR neutron response = self.get(path=path, params=params) [ 923.644723] env[76206]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 923.644723] env[76206]: ERROR neutron raise RuntimeError(msg, last_err) [ 923.644723] env[76206]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 923.644723] env[76206]: ERROR neutron [ 926.324162] env[76309]: INFO neutron.common.config [-] Logging enabled! [ 926.324540] env[76309]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 926.324675] env[76309]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76309) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 926.324900] env[76309]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 926.325044] env[76309]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 926.325880] env[76309]: WARNING oslo_policy.policy [None req-fbf75036-7eb5-40b2-b6eb-3c5fcb1593df None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 926.326075] env[76309]: WARNING oslo_policy.policy [None req-fbf75036-7eb5-40b2-b6eb-3c5fcb1593df None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 926.327385] env[76309]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-fbf75036-7eb5-40b2-b6eb-3c5fcb1593df None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 926.341025] env[76309]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fbf75036-7eb5-40b2-b6eb-3c5fcb1593df None None] Session token - acquiring [ 926.341794] env[76309]: DEBUG networking_nsxv3.common.locking [None req-fbf75036-7eb5-40b2-b6eb-3c5fcb1593df None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76309) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 926.341996] env[76309]: DEBUG oslo_concurrency.lockutils [None req-fbf75036-7eb5-40b2-b6eb-3c5fcb1593df None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76309) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 926.342147] env[76309]: DEBUG oslo_concurrency.lockutils [None req-fbf75036-7eb5-40b2-b6eb-3c5fcb1593df None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76309) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 926.342481] env[76309]: DEBUG oslo_concurrency.lockutils [None req-fbf75036-7eb5-40b2-b6eb-3c5fcb1593df None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76309) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 926.351607] env[76309]: DEBUG oslo_concurrency.lockutils [None req-fbf75036-7eb5-40b2-b6eb-3c5fcb1593df None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76309) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 926.351866] env[76309]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fbf75036-7eb5-40b2-b6eb-3c5fcb1593df None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 926.352048] env[76309]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fbf75036-7eb5-40b2-b6eb-3c5fcb1593df None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76309) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 928.360219] env[76309]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fbf75036-7eb5-40b2-b6eb-3c5fcb1593df None None] Session token - acquiring [ 928.360518] env[76309]: DEBUG networking_nsxv3.common.locking [None req-fbf75036-7eb5-40b2-b6eb-3c5fcb1593df None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76309) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 928.360709] env[76309]: DEBUG oslo_concurrency.lockutils [None req-fbf75036-7eb5-40b2-b6eb-3c5fcb1593df None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76309) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 928.360833] env[76309]: DEBUG oslo_concurrency.lockutils [None req-fbf75036-7eb5-40b2-b6eb-3c5fcb1593df None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76309) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 928.361124] env[76309]: DEBUG oslo_concurrency.lockutils [None req-fbf75036-7eb5-40b2-b6eb-3c5fcb1593df None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76309) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 928.370501] env[76309]: DEBUG oslo_concurrency.lockutils [None req-fbf75036-7eb5-40b2-b6eb-3c5fcb1593df None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76309) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 928.370739] env[76309]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fbf75036-7eb5-40b2-b6eb-3c5fcb1593df None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 928.370877] env[76309]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fbf75036-7eb5-40b2-b6eb-3c5fcb1593df None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76309) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 930.372919] env[76309]: CRITICAL neutron [None req-fbf75036-7eb5-40b2-b6eb-3c5fcb1593df None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 930.372919] env[76309]: ERROR neutron Traceback (most recent call last): [ 930.372919] env[76309]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 930.372919] env[76309]: ERROR neutron sys.exit(main()) [ 930.372919] env[76309]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 930.372919] env[76309]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 930.372919] env[76309]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 930.372919] env[76309]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 930.372919] env[76309]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 930.372919] env[76309]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 930.372919] env[76309]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 930.372919] env[76309]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 930.372919] env[76309]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 930.372919] env[76309]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 930.372919] env[76309]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 930.372919] env[76309]: ERROR neutron response = self.get(path=path, params=params) [ 930.373740] env[76309]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 930.373740] env[76309]: ERROR neutron raise RuntimeError(msg, last_err) [ 930.373740] env[76309]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 930.373740] env[76309]: ERROR neutron [ 933.153872] env[76413]: INFO neutron.common.config [-] Logging enabled! [ 933.154156] env[76413]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 933.154473] env[76413]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76413) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 933.154743] env[76413]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 933.154833] env[76413]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 933.155695] env[76413]: WARNING oslo_policy.policy [None req-13f72885-23e7-45e0-8743-c440aad59134 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 933.155904] env[76413]: WARNING oslo_policy.policy [None req-13f72885-23e7-45e0-8743-c440aad59134 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 933.157198] env[76413]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-13f72885-23e7-45e0-8743-c440aad59134 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 933.171309] env[76413]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-13f72885-23e7-45e0-8743-c440aad59134 None None] Session token - acquiring [ 933.172263] env[76413]: DEBUG networking_nsxv3.common.locking [None req-13f72885-23e7-45e0-8743-c440aad59134 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76413) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 933.172482] env[76413]: DEBUG oslo_concurrency.lockutils [None req-13f72885-23e7-45e0-8743-c440aad59134 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76413) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 933.172560] env[76413]: DEBUG oslo_concurrency.lockutils [None req-13f72885-23e7-45e0-8743-c440aad59134 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76413) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 933.172921] env[76413]: DEBUG oslo_concurrency.lockutils [None req-13f72885-23e7-45e0-8743-c440aad59134 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76413) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 933.182952] env[76413]: DEBUG oslo_concurrency.lockutils [None req-13f72885-23e7-45e0-8743-c440aad59134 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76413) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 933.182952] env[76413]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-13f72885-23e7-45e0-8743-c440aad59134 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 933.182952] env[76413]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-13f72885-23e7-45e0-8743-c440aad59134 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76413) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 935.190527] env[76413]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-13f72885-23e7-45e0-8743-c440aad59134 None None] Session token - acquiring [ 935.190841] env[76413]: DEBUG networking_nsxv3.common.locking [None req-13f72885-23e7-45e0-8743-c440aad59134 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76413) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 935.190998] env[76413]: DEBUG oslo_concurrency.lockutils [None req-13f72885-23e7-45e0-8743-c440aad59134 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76413) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 935.191120] env[76413]: DEBUG oslo_concurrency.lockutils [None req-13f72885-23e7-45e0-8743-c440aad59134 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76413) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 935.191396] env[76413]: DEBUG oslo_concurrency.lockutils [None req-13f72885-23e7-45e0-8743-c440aad59134 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76413) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 935.200729] env[76413]: DEBUG oslo_concurrency.lockutils [None req-13f72885-23e7-45e0-8743-c440aad59134 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76413) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 935.200969] env[76413]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-13f72885-23e7-45e0-8743-c440aad59134 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 935.201104] env[76413]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-13f72885-23e7-45e0-8743-c440aad59134 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76413) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 937.204223] env[76413]: CRITICAL neutron [None req-13f72885-23e7-45e0-8743-c440aad59134 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 937.204223] env[76413]: ERROR neutron Traceback (most recent call last): [ 937.204223] env[76413]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 937.204223] env[76413]: ERROR neutron sys.exit(main()) [ 937.204223] env[76413]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 937.204223] env[76413]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 937.204223] env[76413]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 937.204223] env[76413]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 937.204223] env[76413]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 937.204223] env[76413]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 937.204223] env[76413]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 937.204223] env[76413]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 937.204223] env[76413]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 937.204223] env[76413]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 937.204223] env[76413]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 937.204223] env[76413]: ERROR neutron response = self.get(path=path, params=params) [ 937.205057] env[76413]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 937.205057] env[76413]: ERROR neutron raise RuntimeError(msg, last_err) [ 937.205057] env[76413]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 937.205057] env[76413]: ERROR neutron [ 939.870256] env[76649]: INFO neutron.common.config [-] Logging enabled! [ 939.870594] env[76649]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 939.870805] env[76649]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76649) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 939.871057] env[76649]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 939.871146] env[76649]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 939.872027] env[76649]: WARNING oslo_policy.policy [None req-1b684558-187c-416f-906d-be89e90151e6 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 939.872252] env[76649]: WARNING oslo_policy.policy [None req-1b684558-187c-416f-906d-be89e90151e6 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 939.873500] env[76649]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-1b684558-187c-416f-906d-be89e90151e6 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 939.888582] env[76649]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1b684558-187c-416f-906d-be89e90151e6 None None] Session token - acquiring [ 939.889565] env[76649]: DEBUG networking_nsxv3.common.locking [None req-1b684558-187c-416f-906d-be89e90151e6 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76649) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 939.890073] env[76649]: DEBUG oslo_concurrency.lockutils [None req-1b684558-187c-416f-906d-be89e90151e6 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76649) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 939.890243] env[76649]: DEBUG oslo_concurrency.lockutils [None req-1b684558-187c-416f-906d-be89e90151e6 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76649) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 939.890661] env[76649]: DEBUG oslo_concurrency.lockutils [None req-1b684558-187c-416f-906d-be89e90151e6 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76649) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 939.900467] env[76649]: DEBUG oslo_concurrency.lockutils [None req-1b684558-187c-416f-906d-be89e90151e6 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76649) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 939.900772] env[76649]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1b684558-187c-416f-906d-be89e90151e6 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 939.900962] env[76649]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1b684558-187c-416f-906d-be89e90151e6 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76649) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 941.907009] env[76649]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1b684558-187c-416f-906d-be89e90151e6 None None] Session token - acquiring [ 941.907387] env[76649]: DEBUG networking_nsxv3.common.locking [None req-1b684558-187c-416f-906d-be89e90151e6 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76649) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 941.907618] env[76649]: DEBUG oslo_concurrency.lockutils [None req-1b684558-187c-416f-906d-be89e90151e6 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76649) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 941.907748] env[76649]: DEBUG oslo_concurrency.lockutils [None req-1b684558-187c-416f-906d-be89e90151e6 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76649) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 941.908032] env[76649]: DEBUG oslo_concurrency.lockutils [None req-1b684558-187c-416f-906d-be89e90151e6 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76649) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 941.916907] env[76649]: DEBUG oslo_concurrency.lockutils [None req-1b684558-187c-416f-906d-be89e90151e6 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76649) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 941.917133] env[76649]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1b684558-187c-416f-906d-be89e90151e6 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 941.917271] env[76649]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1b684558-187c-416f-906d-be89e90151e6 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76649) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 943.920124] env[76649]: CRITICAL neutron [None req-1b684558-187c-416f-906d-be89e90151e6 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 943.920124] env[76649]: ERROR neutron Traceback (most recent call last): [ 943.920124] env[76649]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 943.920124] env[76649]: ERROR neutron sys.exit(main()) [ 943.920124] env[76649]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 943.920124] env[76649]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 943.920124] env[76649]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 943.920124] env[76649]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 943.920124] env[76649]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 943.920124] env[76649]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 943.920124] env[76649]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 943.920124] env[76649]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 943.920124] env[76649]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 943.920124] env[76649]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 943.920124] env[76649]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 943.920124] env[76649]: ERROR neutron response = self.get(path=path, params=params) [ 943.920887] env[76649]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 943.920887] env[76649]: ERROR neutron raise RuntimeError(msg, last_err) [ 943.920887] env[76649]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 943.920887] env[76649]: ERROR neutron [ 946.602034] env[76737]: INFO neutron.common.config [-] Logging enabled! [ 946.602402] env[76737]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 946.602684] env[76737]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76737) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 946.603058] env[76737]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 946.603169] env[76737]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 946.604260] env[76737]: WARNING oslo_policy.policy [None req-7e0c56a4-7ab4-4d4c-a1ca-0ca8f6990cff None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 946.604260] env[76737]: WARNING oslo_policy.policy [None req-7e0c56a4-7ab4-4d4c-a1ca-0ca8f6990cff None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 946.605820] env[76737]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-7e0c56a4-7ab4-4d4c-a1ca-0ca8f6990cff None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 946.620259] env[76737]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7e0c56a4-7ab4-4d4c-a1ca-0ca8f6990cff None None] Session token - acquiring [ 946.621410] env[76737]: DEBUG networking_nsxv3.common.locking [None req-7e0c56a4-7ab4-4d4c-a1ca-0ca8f6990cff None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76737) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 946.621533] env[76737]: DEBUG oslo_concurrency.lockutils [None req-7e0c56a4-7ab4-4d4c-a1ca-0ca8f6990cff None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76737) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 946.621661] env[76737]: DEBUG oslo_concurrency.lockutils [None req-7e0c56a4-7ab4-4d4c-a1ca-0ca8f6990cff None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76737) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 946.622072] env[76737]: DEBUG oslo_concurrency.lockutils [None req-7e0c56a4-7ab4-4d4c-a1ca-0ca8f6990cff None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76737) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 946.631775] env[76737]: DEBUG oslo_concurrency.lockutils [None req-7e0c56a4-7ab4-4d4c-a1ca-0ca8f6990cff None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76737) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 946.632141] env[76737]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7e0c56a4-7ab4-4d4c-a1ca-0ca8f6990cff None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 946.632284] env[76737]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7e0c56a4-7ab4-4d4c-a1ca-0ca8f6990cff None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76737) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 948.638325] env[76737]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7e0c56a4-7ab4-4d4c-a1ca-0ca8f6990cff None None] Session token - acquiring [ 948.638621] env[76737]: DEBUG networking_nsxv3.common.locking [None req-7e0c56a4-7ab4-4d4c-a1ca-0ca8f6990cff None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76737) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 948.638783] env[76737]: DEBUG oslo_concurrency.lockutils [None req-7e0c56a4-7ab4-4d4c-a1ca-0ca8f6990cff None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76737) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 948.638912] env[76737]: DEBUG oslo_concurrency.lockutils [None req-7e0c56a4-7ab4-4d4c-a1ca-0ca8f6990cff None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76737) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 948.639186] env[76737]: DEBUG oslo_concurrency.lockutils [None req-7e0c56a4-7ab4-4d4c-a1ca-0ca8f6990cff None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76737) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 948.648271] env[76737]: DEBUG oslo_concurrency.lockutils [None req-7e0c56a4-7ab4-4d4c-a1ca-0ca8f6990cff None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76737) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 948.648500] env[76737]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7e0c56a4-7ab4-4d4c-a1ca-0ca8f6990cff None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 948.648632] env[76737]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7e0c56a4-7ab4-4d4c-a1ca-0ca8f6990cff None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76737) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 950.653581] env[76737]: CRITICAL neutron [None req-7e0c56a4-7ab4-4d4c-a1ca-0ca8f6990cff None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 950.653581] env[76737]: ERROR neutron Traceback (most recent call last): [ 950.653581] env[76737]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 950.653581] env[76737]: ERROR neutron sys.exit(main()) [ 950.653581] env[76737]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 950.653581] env[76737]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 950.653581] env[76737]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 950.653581] env[76737]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 950.653581] env[76737]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 950.653581] env[76737]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 950.653581] env[76737]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 950.653581] env[76737]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 950.653581] env[76737]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 950.653581] env[76737]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 950.653581] env[76737]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 950.653581] env[76737]: ERROR neutron response = self.get(path=path, params=params) [ 950.654306] env[76737]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 950.654306] env[76737]: ERROR neutron raise RuntimeError(msg, last_err) [ 950.654306] env[76737]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 950.654306] env[76737]: ERROR neutron [ 953.311423] env[76983]: INFO neutron.common.config [-] Logging enabled! [ 953.311802] env[76983]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 953.311935] env[76983]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76983) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 953.312228] env[76983]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 953.312358] env[76983]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 953.313181] env[76983]: WARNING oslo_policy.policy [None req-30b9cd1c-2ebe-4618-b963-d35b2ffa81b0 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 953.313417] env[76983]: WARNING oslo_policy.policy [None req-30b9cd1c-2ebe-4618-b963-d35b2ffa81b0 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 953.314623] env[76983]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-30b9cd1c-2ebe-4618-b963-d35b2ffa81b0 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 953.328452] env[76983]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-30b9cd1c-2ebe-4618-b963-d35b2ffa81b0 None None] Session token - acquiring [ 953.329247] env[76983]: DEBUG networking_nsxv3.common.locking [None req-30b9cd1c-2ebe-4618-b963-d35b2ffa81b0 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76983) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 953.329449] env[76983]: DEBUG oslo_concurrency.lockutils [None req-30b9cd1c-2ebe-4618-b963-d35b2ffa81b0 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76983) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 953.329618] env[76983]: DEBUG oslo_concurrency.lockutils [None req-30b9cd1c-2ebe-4618-b963-d35b2ffa81b0 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76983) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 953.329946] env[76983]: DEBUG oslo_concurrency.lockutils [None req-30b9cd1c-2ebe-4618-b963-d35b2ffa81b0 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76983) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 953.338990] env[76983]: DEBUG oslo_concurrency.lockutils [None req-30b9cd1c-2ebe-4618-b963-d35b2ffa81b0 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76983) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 953.339213] env[76983]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-30b9cd1c-2ebe-4618-b963-d35b2ffa81b0 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 953.339369] env[76983]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-30b9cd1c-2ebe-4618-b963-d35b2ffa81b0 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76983) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 955.348520] env[76983]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-30b9cd1c-2ebe-4618-b963-d35b2ffa81b0 None None] Session token - acquiring [ 955.348520] env[76983]: DEBUG networking_nsxv3.common.locking [None req-30b9cd1c-2ebe-4618-b963-d35b2ffa81b0 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76983) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 955.348520] env[76983]: DEBUG oslo_concurrency.lockutils [None req-30b9cd1c-2ebe-4618-b963-d35b2ffa81b0 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76983) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 955.348945] env[76983]: DEBUG oslo_concurrency.lockutils [None req-30b9cd1c-2ebe-4618-b963-d35b2ffa81b0 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76983) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 955.348945] env[76983]: DEBUG oslo_concurrency.lockutils [None req-30b9cd1c-2ebe-4618-b963-d35b2ffa81b0 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76983) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 955.356771] env[76983]: DEBUG oslo_concurrency.lockutils [None req-30b9cd1c-2ebe-4618-b963-d35b2ffa81b0 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76983) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 955.357249] env[76983]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-30b9cd1c-2ebe-4618-b963-d35b2ffa81b0 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 955.357495] env[76983]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-30b9cd1c-2ebe-4618-b963-d35b2ffa81b0 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76983) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 957.360484] env[76983]: CRITICAL neutron [None req-30b9cd1c-2ebe-4618-b963-d35b2ffa81b0 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 957.360484] env[76983]: ERROR neutron Traceback (most recent call last): [ 957.360484] env[76983]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 957.360484] env[76983]: ERROR neutron sys.exit(main()) [ 957.360484] env[76983]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 957.360484] env[76983]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 957.360484] env[76983]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 957.360484] env[76983]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 957.360484] env[76983]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 957.360484] env[76983]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 957.360484] env[76983]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 957.360484] env[76983]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 957.360484] env[76983]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 957.360484] env[76983]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 957.360484] env[76983]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 957.360484] env[76983]: ERROR neutron response = self.get(path=path, params=params) [ 957.361205] env[76983]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 957.361205] env[76983]: ERROR neutron raise RuntimeError(msg, last_err) [ 957.361205] env[76983]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 957.361205] env[76983]: ERROR neutron [ 960.070371] env[77112]: INFO neutron.common.config [-] Logging enabled! [ 960.070790] env[77112]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 960.070871] env[77112]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77112) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 960.071094] env[77112]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 960.071189] env[77112]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 960.071997] env[77112]: WARNING oslo_policy.policy [None req-e5121533-7d72-4dea-8d3c-5430c40ad1c1 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 960.072259] env[77112]: WARNING oslo_policy.policy [None req-e5121533-7d72-4dea-8d3c-5430c40ad1c1 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 960.073456] env[77112]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-e5121533-7d72-4dea-8d3c-5430c40ad1c1 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 960.087896] env[77112]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e5121533-7d72-4dea-8d3c-5430c40ad1c1 None None] Session token - acquiring [ 960.088679] env[77112]: DEBUG networking_nsxv3.common.locking [None req-e5121533-7d72-4dea-8d3c-5430c40ad1c1 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77112) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 960.088888] env[77112]: DEBUG oslo_concurrency.lockutils [None req-e5121533-7d72-4dea-8d3c-5430c40ad1c1 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77112) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 960.089018] env[77112]: DEBUG oslo_concurrency.lockutils [None req-e5121533-7d72-4dea-8d3c-5430c40ad1c1 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77112) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 960.089378] env[77112]: DEBUG oslo_concurrency.lockutils [None req-e5121533-7d72-4dea-8d3c-5430c40ad1c1 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77112) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 960.098296] env[77112]: DEBUG oslo_concurrency.lockutils [None req-e5121533-7d72-4dea-8d3c-5430c40ad1c1 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77112) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 960.098541] env[77112]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e5121533-7d72-4dea-8d3c-5430c40ad1c1 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 960.098689] env[77112]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e5121533-7d72-4dea-8d3c-5430c40ad1c1 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77112) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 962.104952] env[77112]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e5121533-7d72-4dea-8d3c-5430c40ad1c1 None None] Session token - acquiring [ 962.105366] env[77112]: DEBUG networking_nsxv3.common.locking [None req-e5121533-7d72-4dea-8d3c-5430c40ad1c1 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77112) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 962.105522] env[77112]: DEBUG oslo_concurrency.lockutils [None req-e5121533-7d72-4dea-8d3c-5430c40ad1c1 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77112) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 962.105652] env[77112]: DEBUG oslo_concurrency.lockutils [None req-e5121533-7d72-4dea-8d3c-5430c40ad1c1 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77112) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 962.105924] env[77112]: DEBUG oslo_concurrency.lockutils [None req-e5121533-7d72-4dea-8d3c-5430c40ad1c1 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77112) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 962.116750] env[77112]: DEBUG oslo_concurrency.lockutils [None req-e5121533-7d72-4dea-8d3c-5430c40ad1c1 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77112) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 962.117049] env[77112]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e5121533-7d72-4dea-8d3c-5430c40ad1c1 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 962.117237] env[77112]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e5121533-7d72-4dea-8d3c-5430c40ad1c1 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77112) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 964.120183] env[77112]: CRITICAL neutron [None req-e5121533-7d72-4dea-8d3c-5430c40ad1c1 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 964.120183] env[77112]: ERROR neutron Traceback (most recent call last): [ 964.120183] env[77112]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 964.120183] env[77112]: ERROR neutron sys.exit(main()) [ 964.120183] env[77112]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 964.120183] env[77112]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 964.120183] env[77112]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 964.120183] env[77112]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 964.120183] env[77112]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 964.120183] env[77112]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 964.120183] env[77112]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 964.120183] env[77112]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 964.120183] env[77112]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 964.120183] env[77112]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 964.120183] env[77112]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 964.120183] env[77112]: ERROR neutron response = self.get(path=path, params=params) [ 964.121353] env[77112]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 964.121353] env[77112]: ERROR neutron raise RuntimeError(msg, last_err) [ 964.121353] env[77112]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 964.121353] env[77112]: ERROR neutron [ 966.802114] env[77372]: INFO neutron.common.config [-] Logging enabled! [ 966.802455] env[77372]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 966.802680] env[77372]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77372) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 966.802906] env[77372]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 966.803001] env[77372]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 966.803875] env[77372]: WARNING oslo_policy.policy [None req-a656e7af-89f4-41c2-ac2e-ae23bb7a54f1 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 966.804110] env[77372]: WARNING oslo_policy.policy [None req-a656e7af-89f4-41c2-ac2e-ae23bb7a54f1 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 966.805373] env[77372]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-a656e7af-89f4-41c2-ac2e-ae23bb7a54f1 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 966.819836] env[77372]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a656e7af-89f4-41c2-ac2e-ae23bb7a54f1 None None] Session token - acquiring [ 966.820697] env[77372]: DEBUG networking_nsxv3.common.locking [None req-a656e7af-89f4-41c2-ac2e-ae23bb7a54f1 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77372) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 966.820905] env[77372]: DEBUG oslo_concurrency.lockutils [None req-a656e7af-89f4-41c2-ac2e-ae23bb7a54f1 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77372) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 966.821032] env[77372]: DEBUG oslo_concurrency.lockutils [None req-a656e7af-89f4-41c2-ac2e-ae23bb7a54f1 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77372) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 966.821399] env[77372]: DEBUG oslo_concurrency.lockutils [None req-a656e7af-89f4-41c2-ac2e-ae23bb7a54f1 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77372) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 966.830559] env[77372]: DEBUG oslo_concurrency.lockutils [None req-a656e7af-89f4-41c2-ac2e-ae23bb7a54f1 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77372) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 966.830847] env[77372]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a656e7af-89f4-41c2-ac2e-ae23bb7a54f1 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 966.830959] env[77372]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a656e7af-89f4-41c2-ac2e-ae23bb7a54f1 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77372) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 968.837028] env[77372]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a656e7af-89f4-41c2-ac2e-ae23bb7a54f1 None None] Session token - acquiring [ 968.837549] env[77372]: DEBUG networking_nsxv3.common.locking [None req-a656e7af-89f4-41c2-ac2e-ae23bb7a54f1 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77372) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 968.837657] env[77372]: DEBUG oslo_concurrency.lockutils [None req-a656e7af-89f4-41c2-ac2e-ae23bb7a54f1 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77372) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 968.837729] env[77372]: DEBUG oslo_concurrency.lockutils [None req-a656e7af-89f4-41c2-ac2e-ae23bb7a54f1 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77372) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 968.838005] env[77372]: DEBUG oslo_concurrency.lockutils [None req-a656e7af-89f4-41c2-ac2e-ae23bb7a54f1 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77372) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 968.847410] env[77372]: DEBUG oslo_concurrency.lockutils [None req-a656e7af-89f4-41c2-ac2e-ae23bb7a54f1 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77372) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 968.847647] env[77372]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a656e7af-89f4-41c2-ac2e-ae23bb7a54f1 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 968.847784] env[77372]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a656e7af-89f4-41c2-ac2e-ae23bb7a54f1 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77372) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 970.848885] env[77372]: CRITICAL neutron [None req-a656e7af-89f4-41c2-ac2e-ae23bb7a54f1 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 970.848885] env[77372]: ERROR neutron Traceback (most recent call last): [ 970.848885] env[77372]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 970.848885] env[77372]: ERROR neutron sys.exit(main()) [ 970.848885] env[77372]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 970.848885] env[77372]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 970.848885] env[77372]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 970.848885] env[77372]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 970.848885] env[77372]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 970.848885] env[77372]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 970.848885] env[77372]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 970.848885] env[77372]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 970.848885] env[77372]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 970.848885] env[77372]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 970.848885] env[77372]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 970.848885] env[77372]: ERROR neutron response = self.get(path=path, params=params) [ 970.850344] env[77372]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 970.850344] env[77372]: ERROR neutron raise RuntimeError(msg, last_err) [ 970.850344] env[77372]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 970.850344] env[77372]: ERROR neutron [ 973.282717] env[77440]: INFO neutron.common.config [-] Logging enabled! [ 973.283108] env[77440]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 973.283248] env[77440]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77440) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 973.283523] env[77440]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 973.283619] env[77440]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 973.284503] env[77440]: WARNING oslo_policy.policy [None req-0e867210-1431-4d21-8129-b9a3499d3050 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 973.284758] env[77440]: WARNING oslo_policy.policy [None req-0e867210-1431-4d21-8129-b9a3499d3050 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 973.286034] env[77440]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-0e867210-1431-4d21-8129-b9a3499d3050 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 973.299313] env[77440]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0e867210-1431-4d21-8129-b9a3499d3050 None None] Session token - acquiring [ 973.300112] env[77440]: DEBUG networking_nsxv3.common.locking [None req-0e867210-1431-4d21-8129-b9a3499d3050 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77440) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 973.300322] env[77440]: DEBUG oslo_concurrency.lockutils [None req-0e867210-1431-4d21-8129-b9a3499d3050 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77440) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 973.300455] env[77440]: DEBUG oslo_concurrency.lockutils [None req-0e867210-1431-4d21-8129-b9a3499d3050 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77440) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 973.300858] env[77440]: DEBUG oslo_concurrency.lockutils [None req-0e867210-1431-4d21-8129-b9a3499d3050 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77440) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 973.310338] env[77440]: DEBUG oslo_concurrency.lockutils [None req-0e867210-1431-4d21-8129-b9a3499d3050 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77440) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 973.310590] env[77440]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0e867210-1431-4d21-8129-b9a3499d3050 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 973.310741] env[77440]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0e867210-1431-4d21-8129-b9a3499d3050 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77440) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 975.316725] env[77440]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0e867210-1431-4d21-8129-b9a3499d3050 None None] Session token - acquiring [ 975.317038] env[77440]: DEBUG networking_nsxv3.common.locking [None req-0e867210-1431-4d21-8129-b9a3499d3050 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77440) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 975.317536] env[77440]: DEBUG oslo_concurrency.lockutils [None req-0e867210-1431-4d21-8129-b9a3499d3050 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77440) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 975.317536] env[77440]: DEBUG oslo_concurrency.lockutils [None req-0e867210-1431-4d21-8129-b9a3499d3050 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77440) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 975.317675] env[77440]: DEBUG oslo_concurrency.lockutils [None req-0e867210-1431-4d21-8129-b9a3499d3050 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77440) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 975.326590] env[77440]: DEBUG oslo_concurrency.lockutils [None req-0e867210-1431-4d21-8129-b9a3499d3050 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77440) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 975.326846] env[77440]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0e867210-1431-4d21-8129-b9a3499d3050 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 975.326989] env[77440]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0e867210-1431-4d21-8129-b9a3499d3050 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77440) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 977.329900] env[77440]: CRITICAL neutron [None req-0e867210-1431-4d21-8129-b9a3499d3050 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 977.329900] env[77440]: ERROR neutron Traceback (most recent call last): [ 977.329900] env[77440]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 977.329900] env[77440]: ERROR neutron sys.exit(main()) [ 977.329900] env[77440]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 977.329900] env[77440]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 977.329900] env[77440]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 977.329900] env[77440]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 977.329900] env[77440]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 977.329900] env[77440]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 977.329900] env[77440]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 977.329900] env[77440]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 977.329900] env[77440]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 977.329900] env[77440]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 977.329900] env[77440]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 977.329900] env[77440]: ERROR neutron response = self.get(path=path, params=params) [ 977.330855] env[77440]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 977.330855] env[77440]: ERROR neutron raise RuntimeError(msg, last_err) [ 977.330855] env[77440]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 977.330855] env[77440]: ERROR neutron [ 980.075542] env[77545]: INFO neutron.common.config [-] Logging enabled! [ 980.075853] env[77545]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 980.076158] env[77545]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77545) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 980.076411] env[77545]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 980.076505] env[77545]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 980.077345] env[77545]: WARNING oslo_policy.policy [None req-c335dac0-38fe-43ee-80b6-0410306e8574 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 980.077569] env[77545]: WARNING oslo_policy.policy [None req-c335dac0-38fe-43ee-80b6-0410306e8574 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 980.078752] env[77545]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-c335dac0-38fe-43ee-80b6-0410306e8574 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 980.093510] env[77545]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c335dac0-38fe-43ee-80b6-0410306e8574 None None] Session token - acquiring [ 980.094304] env[77545]: DEBUG networking_nsxv3.common.locking [None req-c335dac0-38fe-43ee-80b6-0410306e8574 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77545) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 980.094501] env[77545]: DEBUG oslo_concurrency.lockutils [None req-c335dac0-38fe-43ee-80b6-0410306e8574 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77545) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 980.094631] env[77545]: DEBUG oslo_concurrency.lockutils [None req-c335dac0-38fe-43ee-80b6-0410306e8574 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77545) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 980.094992] env[77545]: DEBUG oslo_concurrency.lockutils [None req-c335dac0-38fe-43ee-80b6-0410306e8574 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77545) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 980.104057] env[77545]: DEBUG oslo_concurrency.lockutils [None req-c335dac0-38fe-43ee-80b6-0410306e8574 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77545) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 980.104308] env[77545]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c335dac0-38fe-43ee-80b6-0410306e8574 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 980.104454] env[77545]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c335dac0-38fe-43ee-80b6-0410306e8574 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77545) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 982.110297] env[77545]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c335dac0-38fe-43ee-80b6-0410306e8574 None None] Session token - acquiring [ 982.110647] env[77545]: DEBUG networking_nsxv3.common.locking [None req-c335dac0-38fe-43ee-80b6-0410306e8574 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77545) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 982.110840] env[77545]: DEBUG oslo_concurrency.lockutils [None req-c335dac0-38fe-43ee-80b6-0410306e8574 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77545) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 982.110965] env[77545]: DEBUG oslo_concurrency.lockutils [None req-c335dac0-38fe-43ee-80b6-0410306e8574 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77545) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 982.111231] env[77545]: DEBUG oslo_concurrency.lockutils [None req-c335dac0-38fe-43ee-80b6-0410306e8574 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77545) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 982.120399] env[77545]: DEBUG oslo_concurrency.lockutils [None req-c335dac0-38fe-43ee-80b6-0410306e8574 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77545) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 982.120664] env[77545]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c335dac0-38fe-43ee-80b6-0410306e8574 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 982.120807] env[77545]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c335dac0-38fe-43ee-80b6-0410306e8574 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77545) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 984.123830] env[77545]: CRITICAL neutron [None req-c335dac0-38fe-43ee-80b6-0410306e8574 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 984.123830] env[77545]: ERROR neutron Traceback (most recent call last): [ 984.123830] env[77545]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 984.123830] env[77545]: ERROR neutron sys.exit(main()) [ 984.123830] env[77545]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 984.123830] env[77545]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 984.123830] env[77545]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 984.123830] env[77545]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 984.123830] env[77545]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 984.123830] env[77545]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 984.123830] env[77545]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 984.123830] env[77545]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 984.123830] env[77545]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 984.123830] env[77545]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 984.123830] env[77545]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 984.123830] env[77545]: ERROR neutron response = self.get(path=path, params=params) [ 984.125421] env[77545]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 984.125421] env[77545]: ERROR neutron raise RuntimeError(msg, last_err) [ 984.125421] env[77545]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 984.125421] env[77545]: ERROR neutron [ 986.533739] env[77624]: INFO neutron.common.config [-] Logging enabled! [ 986.534146] env[77624]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 986.534368] env[77624]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77624) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 986.534641] env[77624]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 986.534757] env[77624]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 986.535723] env[77624]: WARNING oslo_policy.policy [None req-132067dd-b82e-4057-b990-de2fbdd3d7b3 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 986.536059] env[77624]: WARNING oslo_policy.policy [None req-132067dd-b82e-4057-b990-de2fbdd3d7b3 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 986.537882] env[77624]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-132067dd-b82e-4057-b990-de2fbdd3d7b3 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 986.555981] env[77624]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-132067dd-b82e-4057-b990-de2fbdd3d7b3 None None] Session token - acquiring [ 986.558546] env[77624]: DEBUG networking_nsxv3.common.locking [None req-132067dd-b82e-4057-b990-de2fbdd3d7b3 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77624) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 986.558546] env[77624]: DEBUG oslo_concurrency.lockutils [None req-132067dd-b82e-4057-b990-de2fbdd3d7b3 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77624) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 986.558802] env[77624]: DEBUG oslo_concurrency.lockutils [None req-132067dd-b82e-4057-b990-de2fbdd3d7b3 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77624) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 986.558802] env[77624]: DEBUG oslo_concurrency.lockutils [None req-132067dd-b82e-4057-b990-de2fbdd3d7b3 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77624) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 986.568876] env[77624]: DEBUG oslo_concurrency.lockutils [None req-132067dd-b82e-4057-b990-de2fbdd3d7b3 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77624) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 986.569232] env[77624]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-132067dd-b82e-4057-b990-de2fbdd3d7b3 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 986.569488] env[77624]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-132067dd-b82e-4057-b990-de2fbdd3d7b3 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77624) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 988.578086] env[77624]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-132067dd-b82e-4057-b990-de2fbdd3d7b3 None None] Session token - acquiring [ 988.578409] env[77624]: DEBUG networking_nsxv3.common.locking [None req-132067dd-b82e-4057-b990-de2fbdd3d7b3 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77624) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 988.578602] env[77624]: DEBUG oslo_concurrency.lockutils [None req-132067dd-b82e-4057-b990-de2fbdd3d7b3 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77624) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 988.578726] env[77624]: DEBUG oslo_concurrency.lockutils [None req-132067dd-b82e-4057-b990-de2fbdd3d7b3 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77624) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 988.578998] env[77624]: DEBUG oslo_concurrency.lockutils [None req-132067dd-b82e-4057-b990-de2fbdd3d7b3 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77624) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 988.587932] env[77624]: DEBUG oslo_concurrency.lockutils [None req-132067dd-b82e-4057-b990-de2fbdd3d7b3 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77624) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 988.588182] env[77624]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-132067dd-b82e-4057-b990-de2fbdd3d7b3 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 988.588324] env[77624]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-132067dd-b82e-4057-b990-de2fbdd3d7b3 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77624) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 990.591409] env[77624]: CRITICAL neutron [None req-132067dd-b82e-4057-b990-de2fbdd3d7b3 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 990.591409] env[77624]: ERROR neutron Traceback (most recent call last): [ 990.591409] env[77624]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 990.591409] env[77624]: ERROR neutron sys.exit(main()) [ 990.591409] env[77624]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 990.591409] env[77624]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 990.591409] env[77624]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 990.591409] env[77624]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 990.591409] env[77624]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 990.591409] env[77624]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 990.591409] env[77624]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 990.591409] env[77624]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 990.591409] env[77624]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 990.591409] env[77624]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 990.591409] env[77624]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 990.591409] env[77624]: ERROR neutron response = self.get(path=path, params=params) [ 990.592264] env[77624]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 990.592264] env[77624]: ERROR neutron raise RuntimeError(msg, last_err) [ 990.592264] env[77624]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 990.592264] env[77624]: ERROR neutron [ 993.152058] env[77696]: INFO neutron.common.config [-] Logging enabled! [ 993.152381] env[77696]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 993.152637] env[77696]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77696) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 993.152839] env[77696]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 993.152942] env[77696]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 993.154140] env[77696]: WARNING oslo_policy.policy [None req-d9b4002c-aded-4baa-b2a0-f02825e9fa0f None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 993.154140] env[77696]: WARNING oslo_policy.policy [None req-d9b4002c-aded-4baa-b2a0-f02825e9fa0f None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 993.155312] env[77696]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-d9b4002c-aded-4baa-b2a0-f02825e9fa0f None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 993.169568] env[77696]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d9b4002c-aded-4baa-b2a0-f02825e9fa0f None None] Session token - acquiring [ 993.170365] env[77696]: DEBUG networking_nsxv3.common.locking [None req-d9b4002c-aded-4baa-b2a0-f02825e9fa0f None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77696) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 993.170569] env[77696]: DEBUG oslo_concurrency.lockutils [None req-d9b4002c-aded-4baa-b2a0-f02825e9fa0f None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77696) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 993.170696] env[77696]: DEBUG oslo_concurrency.lockutils [None req-d9b4002c-aded-4baa-b2a0-f02825e9fa0f None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77696) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 993.171061] env[77696]: DEBUG oslo_concurrency.lockutils [None req-d9b4002c-aded-4baa-b2a0-f02825e9fa0f None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77696) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 993.180166] env[77696]: DEBUG oslo_concurrency.lockutils [None req-d9b4002c-aded-4baa-b2a0-f02825e9fa0f None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77696) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 993.180427] env[77696]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d9b4002c-aded-4baa-b2a0-f02825e9fa0f None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 993.180580] env[77696]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d9b4002c-aded-4baa-b2a0-f02825e9fa0f None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77696) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 995.186776] env[77696]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d9b4002c-aded-4baa-b2a0-f02825e9fa0f None None] Session token - acquiring [ 995.187033] env[77696]: DEBUG networking_nsxv3.common.locking [None req-d9b4002c-aded-4baa-b2a0-f02825e9fa0f None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77696) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 995.187111] env[77696]: DEBUG oslo_concurrency.lockutils [None req-d9b4002c-aded-4baa-b2a0-f02825e9fa0f None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77696) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 995.187177] env[77696]: DEBUG oslo_concurrency.lockutils [None req-d9b4002c-aded-4baa-b2a0-f02825e9fa0f None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77696) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 995.187453] env[77696]: DEBUG oslo_concurrency.lockutils [None req-d9b4002c-aded-4baa-b2a0-f02825e9fa0f None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77696) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 995.197044] env[77696]: DEBUG oslo_concurrency.lockutils [None req-d9b4002c-aded-4baa-b2a0-f02825e9fa0f None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77696) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 995.197295] env[77696]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d9b4002c-aded-4baa-b2a0-f02825e9fa0f None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 995.197425] env[77696]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d9b4002c-aded-4baa-b2a0-f02825e9fa0f None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77696) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 997.199148] env[77696]: CRITICAL neutron [None req-d9b4002c-aded-4baa-b2a0-f02825e9fa0f None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 997.199148] env[77696]: ERROR neutron Traceback (most recent call last): [ 997.199148] env[77696]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 997.199148] env[77696]: ERROR neutron sys.exit(main()) [ 997.199148] env[77696]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 997.199148] env[77696]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 997.199148] env[77696]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 997.199148] env[77696]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 997.199148] env[77696]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 997.199148] env[77696]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 997.199148] env[77696]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 997.199148] env[77696]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 997.199148] env[77696]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 997.199148] env[77696]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 997.199148] env[77696]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 997.199148] env[77696]: ERROR neutron response = self.get(path=path, params=params) [ 997.199951] env[77696]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 997.199951] env[77696]: ERROR neutron raise RuntimeError(msg, last_err) [ 997.199951] env[77696]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 997.199951] env[77696]: ERROR neutron [ 999.922015] env[77926]: INFO neutron.common.config [-] Logging enabled! [ 999.922423] env[77926]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 999.922639] env[77926]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77926) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 999.922782] env[77926]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 999.922960] env[77926]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 999.923723] env[77926]: WARNING oslo_policy.policy [None req-1c657aec-0304-4d82-a5ec-ec55898c6398 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 999.923941] env[77926]: WARNING oslo_policy.policy [None req-1c657aec-0304-4d82-a5ec-ec55898c6398 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 999.925174] env[77926]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-1c657aec-0304-4d82-a5ec-ec55898c6398 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 999.940144] env[77926]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1c657aec-0304-4d82-a5ec-ec55898c6398 None None] Session token - acquiring [ 999.940825] env[77926]: DEBUG networking_nsxv3.common.locking [None req-1c657aec-0304-4d82-a5ec-ec55898c6398 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77926) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 999.941036] env[77926]: DEBUG oslo_concurrency.lockutils [None req-1c657aec-0304-4d82-a5ec-ec55898c6398 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77926) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 999.941190] env[77926]: DEBUG oslo_concurrency.lockutils [None req-1c657aec-0304-4d82-a5ec-ec55898c6398 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77926) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 999.941716] env[77926]: DEBUG oslo_concurrency.lockutils [None req-1c657aec-0304-4d82-a5ec-ec55898c6398 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77926) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 999.951675] env[77926]: DEBUG oslo_concurrency.lockutils [None req-1c657aec-0304-4d82-a5ec-ec55898c6398 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77926) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 999.951940] env[77926]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1c657aec-0304-4d82-a5ec-ec55898c6398 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 999.952105] env[77926]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1c657aec-0304-4d82-a5ec-ec55898c6398 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77926) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1001.958206] env[77926]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1c657aec-0304-4d82-a5ec-ec55898c6398 None None] Session token - acquiring [ 1001.958516] env[77926]: DEBUG networking_nsxv3.common.locking [None req-1c657aec-0304-4d82-a5ec-ec55898c6398 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77926) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1001.958673] env[77926]: DEBUG oslo_concurrency.lockutils [None req-1c657aec-0304-4d82-a5ec-ec55898c6398 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77926) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1001.958795] env[77926]: DEBUG oslo_concurrency.lockutils [None req-1c657aec-0304-4d82-a5ec-ec55898c6398 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77926) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1001.959071] env[77926]: DEBUG oslo_concurrency.lockutils [None req-1c657aec-0304-4d82-a5ec-ec55898c6398 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77926) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1001.967844] env[77926]: DEBUG oslo_concurrency.lockutils [None req-1c657aec-0304-4d82-a5ec-ec55898c6398 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77926) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1001.968064] env[77926]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1c657aec-0304-4d82-a5ec-ec55898c6398 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1001.968877] env[77926]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1c657aec-0304-4d82-a5ec-ec55898c6398 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77926) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1003.969825] env[77926]: CRITICAL neutron [None req-1c657aec-0304-4d82-a5ec-ec55898c6398 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1003.969825] env[77926]: ERROR neutron Traceback (most recent call last): [ 1003.969825] env[77926]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 1003.969825] env[77926]: ERROR neutron sys.exit(main()) [ 1003.969825] env[77926]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 1003.969825] env[77926]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 1003.969825] env[77926]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 1003.969825] env[77926]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 1003.969825] env[77926]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 1003.969825] env[77926]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 1003.969825] env[77926]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 1003.969825] env[77926]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 1003.969825] env[77926]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 1003.969825] env[77926]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 1003.969825] env[77926]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 1003.969825] env[77926]: ERROR neutron response = self.get(path=path, params=params) [ 1003.970757] env[77926]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 1003.970757] env[77926]: ERROR neutron raise RuntimeError(msg, last_err) [ 1003.970757] env[77926]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1003.970757] env[77926]: ERROR neutron [ 1006.535708] env[78171]: INFO neutron.common.config [-] Logging enabled! [ 1006.536048] env[78171]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 1006.536295] env[78171]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=78171) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 1006.536547] env[78171]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 1006.536640] env[78171]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 1006.537471] env[78171]: WARNING oslo_policy.policy [None req-7d5bbc72-528e-48ec-96af-89552bf904ef None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1006.537786] env[78171]: WARNING oslo_policy.policy [None req-7d5bbc72-528e-48ec-96af-89552bf904ef None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1006.539015] env[78171]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-7d5bbc72-528e-48ec-96af-89552bf904ef None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 1006.553980] env[78171]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7d5bbc72-528e-48ec-96af-89552bf904ef None None] Session token - acquiring [ 1006.555294] env[78171]: DEBUG networking_nsxv3.common.locking [None req-7d5bbc72-528e-48ec-96af-89552bf904ef None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=78171) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1006.555506] env[78171]: DEBUG oslo_concurrency.lockutils [None req-7d5bbc72-528e-48ec-96af-89552bf904ef None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78171) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1006.555634] env[78171]: DEBUG oslo_concurrency.lockutils [None req-7d5bbc72-528e-48ec-96af-89552bf904ef None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78171) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1006.556100] env[78171]: DEBUG oslo_concurrency.lockutils [None req-7d5bbc72-528e-48ec-96af-89552bf904ef None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78171) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1006.565224] env[78171]: DEBUG oslo_concurrency.lockutils [None req-7d5bbc72-528e-48ec-96af-89552bf904ef None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78171) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1006.565479] env[78171]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7d5bbc72-528e-48ec-96af-89552bf904ef None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1006.565629] env[78171]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7d5bbc72-528e-48ec-96af-89552bf904ef None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=78171) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1008.571789] env[78171]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7d5bbc72-528e-48ec-96af-89552bf904ef None None] Session token - acquiring [ 1008.572185] env[78171]: DEBUG networking_nsxv3.common.locking [None req-7d5bbc72-528e-48ec-96af-89552bf904ef None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=78171) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1008.572269] env[78171]: DEBUG oslo_concurrency.lockutils [None req-7d5bbc72-528e-48ec-96af-89552bf904ef None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78171) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1008.572367] env[78171]: DEBUG oslo_concurrency.lockutils [None req-7d5bbc72-528e-48ec-96af-89552bf904ef None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78171) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1008.572896] env[78171]: DEBUG oslo_concurrency.lockutils [None req-7d5bbc72-528e-48ec-96af-89552bf904ef None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78171) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1008.581915] env[78171]: DEBUG oslo_concurrency.lockutils [None req-7d5bbc72-528e-48ec-96af-89552bf904ef None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78171) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1008.582164] env[78171]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7d5bbc72-528e-48ec-96af-89552bf904ef None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1008.582306] env[78171]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7d5bbc72-528e-48ec-96af-89552bf904ef None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=78171) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1010.584966] env[78171]: CRITICAL neutron [None req-7d5bbc72-528e-48ec-96af-89552bf904ef None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1010.584966] env[78171]: ERROR neutron Traceback (most recent call last): [ 1010.584966] env[78171]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 1010.584966] env[78171]: ERROR neutron sys.exit(main()) [ 1010.584966] env[78171]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 1010.584966] env[78171]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 1010.584966] env[78171]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 1010.584966] env[78171]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 1010.584966] env[78171]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 1010.584966] env[78171]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 1010.584966] env[78171]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 1010.584966] env[78171]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 1010.584966] env[78171]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 1010.584966] env[78171]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 1010.584966] env[78171]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 1010.584966] env[78171]: ERROR neutron response = self.get(path=path, params=params) [ 1010.585934] env[78171]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 1010.585934] env[78171]: ERROR neutron raise RuntimeError(msg, last_err) [ 1010.585934] env[78171]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1010.585934] env[78171]: ERROR neutron [ 1012.976707] env[78214]: INFO neutron.common.config [-] Logging enabled! [ 1012.977043] env[78214]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 1012.977329] env[78214]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=78214) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 1012.977558] env[78214]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 1012.977652] env[78214]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 1012.978470] env[78214]: WARNING oslo_policy.policy [None req-558400b8-8ed5-4a15-974d-90bfa080e54f None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1012.978695] env[78214]: WARNING oslo_policy.policy [None req-558400b8-8ed5-4a15-974d-90bfa080e54f None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1012.979887] env[78214]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-558400b8-8ed5-4a15-974d-90bfa080e54f None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 1012.994335] env[78214]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-558400b8-8ed5-4a15-974d-90bfa080e54f None None] Session token - acquiring [ 1012.995185] env[78214]: DEBUG networking_nsxv3.common.locking [None req-558400b8-8ed5-4a15-974d-90bfa080e54f None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=78214) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1012.995423] env[78214]: DEBUG oslo_concurrency.lockutils [None req-558400b8-8ed5-4a15-974d-90bfa080e54f None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78214) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1012.995579] env[78214]: DEBUG oslo_concurrency.lockutils [None req-558400b8-8ed5-4a15-974d-90bfa080e54f None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78214) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1012.995975] env[78214]: DEBUG oslo_concurrency.lockutils [None req-558400b8-8ed5-4a15-974d-90bfa080e54f None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78214) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1013.005163] env[78214]: DEBUG oslo_concurrency.lockutils [None req-558400b8-8ed5-4a15-974d-90bfa080e54f None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78214) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1013.005534] env[78214]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-558400b8-8ed5-4a15-974d-90bfa080e54f None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1013.005634] env[78214]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-558400b8-8ed5-4a15-974d-90bfa080e54f None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=78214) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1015.011891] env[78214]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-558400b8-8ed5-4a15-974d-90bfa080e54f None None] Session token - acquiring [ 1015.012272] env[78214]: DEBUG networking_nsxv3.common.locking [None req-558400b8-8ed5-4a15-974d-90bfa080e54f None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=78214) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1015.012344] env[78214]: DEBUG oslo_concurrency.lockutils [None req-558400b8-8ed5-4a15-974d-90bfa080e54f None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78214) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1015.012451] env[78214]: DEBUG oslo_concurrency.lockutils [None req-558400b8-8ed5-4a15-974d-90bfa080e54f None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78214) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1015.012723] env[78214]: DEBUG oslo_concurrency.lockutils [None req-558400b8-8ed5-4a15-974d-90bfa080e54f None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78214) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1015.021923] env[78214]: DEBUG oslo_concurrency.lockutils [None req-558400b8-8ed5-4a15-974d-90bfa080e54f None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78214) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1015.022174] env[78214]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-558400b8-8ed5-4a15-974d-90bfa080e54f None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1015.022329] env[78214]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-558400b8-8ed5-4a15-974d-90bfa080e54f None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=78214) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1017.024882] env[78214]: CRITICAL neutron [None req-558400b8-8ed5-4a15-974d-90bfa080e54f None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1017.024882] env[78214]: ERROR neutron Traceback (most recent call last): [ 1017.024882] env[78214]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 1017.024882] env[78214]: ERROR neutron sys.exit(main()) [ 1017.024882] env[78214]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 1017.024882] env[78214]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 1017.024882] env[78214]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 1017.024882] env[78214]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 1017.024882] env[78214]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 1017.024882] env[78214]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 1017.024882] env[78214]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 1017.024882] env[78214]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 1017.024882] env[78214]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 1017.024882] env[78214]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 1017.024882] env[78214]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 1017.024882] env[78214]: ERROR neutron response = self.get(path=path, params=params) [ 1017.025821] env[78214]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 1017.025821] env[78214]: ERROR neutron raise RuntimeError(msg, last_err) [ 1017.025821] env[78214]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1017.025821] env[78214]: ERROR neutron [ 1019.503427] env[78247]: INFO neutron.common.config [-] Logging enabled! [ 1019.503890] env[78247]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 1019.503944] env[78247]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=78247) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 1019.504209] env[78247]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 1019.504307] env[78247]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 1019.505129] env[78247]: WARNING oslo_policy.policy [None req-2cf418a2-1b8c-429b-9251-a386b5c093f7 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1019.505360] env[78247]: WARNING oslo_policy.policy [None req-2cf418a2-1b8c-429b-9251-a386b5c093f7 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1019.506539] env[78247]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-2cf418a2-1b8c-429b-9251-a386b5c093f7 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 1019.520489] env[78247]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2cf418a2-1b8c-429b-9251-a386b5c093f7 None None] Session token - acquiring [ 1019.521237] env[78247]: DEBUG networking_nsxv3.common.locking [None req-2cf418a2-1b8c-429b-9251-a386b5c093f7 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=78247) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1019.521420] env[78247]: DEBUG oslo_concurrency.lockutils [None req-2cf418a2-1b8c-429b-9251-a386b5c093f7 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78247) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1019.521547] env[78247]: DEBUG oslo_concurrency.lockutils [None req-2cf418a2-1b8c-429b-9251-a386b5c093f7 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78247) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1019.521893] env[78247]: DEBUG oslo_concurrency.lockutils [None req-2cf418a2-1b8c-429b-9251-a386b5c093f7 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78247) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1019.530732] env[78247]: DEBUG oslo_concurrency.lockutils [None req-2cf418a2-1b8c-429b-9251-a386b5c093f7 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78247) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1019.530971] env[78247]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2cf418a2-1b8c-429b-9251-a386b5c093f7 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1019.531120] env[78247]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2cf418a2-1b8c-429b-9251-a386b5c093f7 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=78247) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1021.537979] env[78247]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2cf418a2-1b8c-429b-9251-a386b5c093f7 None None] Session token - acquiring [ 1021.538372] env[78247]: DEBUG networking_nsxv3.common.locking [None req-2cf418a2-1b8c-429b-9251-a386b5c093f7 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=78247) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1021.538447] env[78247]: DEBUG oslo_concurrency.lockutils [None req-2cf418a2-1b8c-429b-9251-a386b5c093f7 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78247) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1021.538576] env[78247]: DEBUG oslo_concurrency.lockutils [None req-2cf418a2-1b8c-429b-9251-a386b5c093f7 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78247) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1021.538839] env[78247]: DEBUG oslo_concurrency.lockutils [None req-2cf418a2-1b8c-429b-9251-a386b5c093f7 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78247) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1021.547792] env[78247]: DEBUG oslo_concurrency.lockutils [None req-2cf418a2-1b8c-429b-9251-a386b5c093f7 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78247) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1021.548030] env[78247]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2cf418a2-1b8c-429b-9251-a386b5c093f7 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1021.548176] env[78247]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2cf418a2-1b8c-429b-9251-a386b5c093f7 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=78247) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1023.549740] env[78247]: CRITICAL neutron [None req-2cf418a2-1b8c-429b-9251-a386b5c093f7 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1023.549740] env[78247]: ERROR neutron Traceback (most recent call last): [ 1023.549740] env[78247]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 1023.549740] env[78247]: ERROR neutron sys.exit(main()) [ 1023.549740] env[78247]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 1023.549740] env[78247]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 1023.549740] env[78247]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 1023.549740] env[78247]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 1023.549740] env[78247]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 1023.549740] env[78247]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 1023.549740] env[78247]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 1023.549740] env[78247]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 1023.549740] env[78247]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 1023.549740] env[78247]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 1023.549740] env[78247]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 1023.549740] env[78247]: ERROR neutron response = self.get(path=path, params=params) [ 1023.550477] env[78247]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 1023.550477] env[78247]: ERROR neutron raise RuntimeError(msg, last_err) [ 1023.550477] env[78247]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1023.550477] env[78247]: ERROR neutron [ 1025.954688] env[78334]: INFO neutron.common.config [-] Logging enabled! [ 1025.955207] env[78334]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 1025.955284] env[78334]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=78334) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 1025.955457] env[78334]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 1025.955552] env[78334]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 1025.956459] env[78334]: WARNING oslo_policy.policy [None req-085b0823-1466-4066-b6ac-29f8101bb0de None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1025.956671] env[78334]: WARNING oslo_policy.policy [None req-085b0823-1466-4066-b6ac-29f8101bb0de None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1025.957829] env[78334]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-085b0823-1466-4066-b6ac-29f8101bb0de None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 1025.971977] env[78334]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-085b0823-1466-4066-b6ac-29f8101bb0de None None] Session token - acquiring [ 1025.972758] env[78334]: DEBUG networking_nsxv3.common.locking [None req-085b0823-1466-4066-b6ac-29f8101bb0de None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=78334) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1025.972962] env[78334]: DEBUG oslo_concurrency.lockutils [None req-085b0823-1466-4066-b6ac-29f8101bb0de None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78334) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1025.973089] env[78334]: DEBUG oslo_concurrency.lockutils [None req-085b0823-1466-4066-b6ac-29f8101bb0de None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78334) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1025.973444] env[78334]: DEBUG oslo_concurrency.lockutils [None req-085b0823-1466-4066-b6ac-29f8101bb0de None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78334) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1025.981880] env[78334]: DEBUG oslo_concurrency.lockutils [None req-085b0823-1466-4066-b6ac-29f8101bb0de None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78334) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1025.982120] env[78334]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-085b0823-1466-4066-b6ac-29f8101bb0de None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1025.982267] env[78334]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-085b0823-1466-4066-b6ac-29f8101bb0de None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=78334) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1027.988731] env[78334]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-085b0823-1466-4066-b6ac-29f8101bb0de None None] Session token - acquiring [ 1027.989267] env[78334]: DEBUG networking_nsxv3.common.locking [None req-085b0823-1466-4066-b6ac-29f8101bb0de None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=78334) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1027.989382] env[78334]: DEBUG oslo_concurrency.lockutils [None req-085b0823-1466-4066-b6ac-29f8101bb0de None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78334) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1027.989493] env[78334]: DEBUG oslo_concurrency.lockutils [None req-085b0823-1466-4066-b6ac-29f8101bb0de None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78334) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1027.989795] env[78334]: DEBUG oslo_concurrency.lockutils [None req-085b0823-1466-4066-b6ac-29f8101bb0de None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78334) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1027.998941] env[78334]: DEBUG oslo_concurrency.lockutils [None req-085b0823-1466-4066-b6ac-29f8101bb0de None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78334) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1027.999179] env[78334]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-085b0823-1466-4066-b6ac-29f8101bb0de None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1027.999318] env[78334]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-085b0823-1466-4066-b6ac-29f8101bb0de None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=78334) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1030.002367] env[78334]: CRITICAL neutron [None req-085b0823-1466-4066-b6ac-29f8101bb0de None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1030.002367] env[78334]: ERROR neutron Traceback (most recent call last): [ 1030.002367] env[78334]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 1030.002367] env[78334]: ERROR neutron sys.exit(main()) [ 1030.002367] env[78334]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 1030.002367] env[78334]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 1030.002367] env[78334]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 1030.002367] env[78334]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 1030.002367] env[78334]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 1030.002367] env[78334]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 1030.002367] env[78334]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 1030.002367] env[78334]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 1030.002367] env[78334]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 1030.002367] env[78334]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 1030.002367] env[78334]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 1030.002367] env[78334]: ERROR neutron response = self.get(path=path, params=params) [ 1030.003775] env[78334]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 1030.003775] env[78334]: ERROR neutron raise RuntimeError(msg, last_err) [ 1030.003775] env[78334]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1030.003775] env[78334]: ERROR neutron [ 1032.443121] env[78341]: INFO neutron.common.config [-] Logging enabled! [ 1032.443548] env[78341]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 1032.443593] env[78341]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=78341) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 1032.443850] env[78341]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 1032.443963] env[78341]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 1032.444809] env[78341]: WARNING oslo_policy.policy [None req-2d6e1550-af13-4b26-b6ba-adef7b08d9b1 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1032.445057] env[78341]: WARNING oslo_policy.policy [None req-2d6e1550-af13-4b26-b6ba-adef7b08d9b1 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1032.446256] env[78341]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-2d6e1550-af13-4b26-b6ba-adef7b08d9b1 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 1032.460640] env[78341]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2d6e1550-af13-4b26-b6ba-adef7b08d9b1 None None] Session token - acquiring [ 1032.461421] env[78341]: DEBUG networking_nsxv3.common.locking [None req-2d6e1550-af13-4b26-b6ba-adef7b08d9b1 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=78341) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1032.461621] env[78341]: DEBUG oslo_concurrency.lockutils [None req-2d6e1550-af13-4b26-b6ba-adef7b08d9b1 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78341) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1032.461744] env[78341]: DEBUG oslo_concurrency.lockutils [None req-2d6e1550-af13-4b26-b6ba-adef7b08d9b1 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78341) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1032.462103] env[78341]: DEBUG oslo_concurrency.lockutils [None req-2d6e1550-af13-4b26-b6ba-adef7b08d9b1 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78341) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1032.471012] env[78341]: DEBUG oslo_concurrency.lockutils [None req-2d6e1550-af13-4b26-b6ba-adef7b08d9b1 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78341) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1032.471258] env[78341]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2d6e1550-af13-4b26-b6ba-adef7b08d9b1 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1032.471412] env[78341]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2d6e1550-af13-4b26-b6ba-adef7b08d9b1 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=78341) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1034.477595] env[78341]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2d6e1550-af13-4b26-b6ba-adef7b08d9b1 None None] Session token - acquiring [ 1034.478077] env[78341]: DEBUG networking_nsxv3.common.locking [None req-2d6e1550-af13-4b26-b6ba-adef7b08d9b1 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=78341) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1034.478304] env[78341]: DEBUG oslo_concurrency.lockutils [None req-2d6e1550-af13-4b26-b6ba-adef7b08d9b1 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78341) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1034.478427] env[78341]: DEBUG oslo_concurrency.lockutils [None req-2d6e1550-af13-4b26-b6ba-adef7b08d9b1 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78341) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1034.478680] env[78341]: DEBUG oslo_concurrency.lockutils [None req-2d6e1550-af13-4b26-b6ba-adef7b08d9b1 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78341) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1034.487921] env[78341]: DEBUG oslo_concurrency.lockutils [None req-2d6e1550-af13-4b26-b6ba-adef7b08d9b1 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78341) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1034.488168] env[78341]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2d6e1550-af13-4b26-b6ba-adef7b08d9b1 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1034.488314] env[78341]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2d6e1550-af13-4b26-b6ba-adef7b08d9b1 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=78341) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1036.491293] env[78341]: CRITICAL neutron [None req-2d6e1550-af13-4b26-b6ba-adef7b08d9b1 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1036.491293] env[78341]: ERROR neutron Traceback (most recent call last): [ 1036.491293] env[78341]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 1036.491293] env[78341]: ERROR neutron sys.exit(main()) [ 1036.491293] env[78341]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 1036.491293] env[78341]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 1036.491293] env[78341]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 1036.491293] env[78341]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 1036.491293] env[78341]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 1036.491293] env[78341]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 1036.491293] env[78341]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 1036.491293] env[78341]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 1036.491293] env[78341]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 1036.491293] env[78341]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 1036.491293] env[78341]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 1036.491293] env[78341]: ERROR neutron response = self.get(path=path, params=params) [ 1036.492172] env[78341]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 1036.492172] env[78341]: ERROR neutron raise RuntimeError(msg, last_err) [ 1036.492172] env[78341]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1036.492172] env[78341]: ERROR neutron [ 1038.932129] env[78392]: INFO neutron.common.config [-] Logging enabled! [ 1038.932553] env[78392]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 1038.932934] env[78392]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=78392) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 1038.933160] env[78392]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 1038.933259] env[78392]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 1038.934238] env[78392]: WARNING oslo_policy.policy [None req-f008d3ac-d0d2-4ffa-949f-f93667ebbbdd None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1038.934457] env[78392]: WARNING oslo_policy.policy [None req-f008d3ac-d0d2-4ffa-949f-f93667ebbbdd None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1038.935652] env[78392]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-f008d3ac-d0d2-4ffa-949f-f93667ebbbdd None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 1038.949381] env[78392]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f008d3ac-d0d2-4ffa-949f-f93667ebbbdd None None] Session token - acquiring [ 1038.950127] env[78392]: DEBUG networking_nsxv3.common.locking [None req-f008d3ac-d0d2-4ffa-949f-f93667ebbbdd None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=78392) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1038.950324] env[78392]: DEBUG oslo_concurrency.lockutils [None req-f008d3ac-d0d2-4ffa-949f-f93667ebbbdd None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78392) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1038.950449] env[78392]: DEBUG oslo_concurrency.lockutils [None req-f008d3ac-d0d2-4ffa-949f-f93667ebbbdd None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78392) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1038.950799] env[78392]: DEBUG oslo_concurrency.lockutils [None req-f008d3ac-d0d2-4ffa-949f-f93667ebbbdd None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78392) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1038.959411] env[78392]: DEBUG oslo_concurrency.lockutils [None req-f008d3ac-d0d2-4ffa-949f-f93667ebbbdd None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78392) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1038.959652] env[78392]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f008d3ac-d0d2-4ffa-949f-f93667ebbbdd None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1038.959799] env[78392]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f008d3ac-d0d2-4ffa-949f-f93667ebbbdd None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=78392) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1040.967974] env[78392]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f008d3ac-d0d2-4ffa-949f-f93667ebbbdd None None] Session token - acquiring [ 1040.968406] env[78392]: DEBUG networking_nsxv3.common.locking [None req-f008d3ac-d0d2-4ffa-949f-f93667ebbbdd None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=78392) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1040.968575] env[78392]: DEBUG oslo_concurrency.lockutils [None req-f008d3ac-d0d2-4ffa-949f-f93667ebbbdd None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78392) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1040.968702] env[78392]: DEBUG oslo_concurrency.lockutils [None req-f008d3ac-d0d2-4ffa-949f-f93667ebbbdd None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78392) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1040.968988] env[78392]: DEBUG oslo_concurrency.lockutils [None req-f008d3ac-d0d2-4ffa-949f-f93667ebbbdd None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78392) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1040.978081] env[78392]: DEBUG oslo_concurrency.lockutils [None req-f008d3ac-d0d2-4ffa-949f-f93667ebbbdd None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78392) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1040.978313] env[78392]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f008d3ac-d0d2-4ffa-949f-f93667ebbbdd None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1040.978451] env[78392]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f008d3ac-d0d2-4ffa-949f-f93667ebbbdd None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=78392) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1042.981414] env[78392]: CRITICAL neutron [None req-f008d3ac-d0d2-4ffa-949f-f93667ebbbdd None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1042.981414] env[78392]: ERROR neutron Traceback (most recent call last): [ 1042.981414] env[78392]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 1042.981414] env[78392]: ERROR neutron sys.exit(main()) [ 1042.981414] env[78392]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 1042.981414] env[78392]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 1042.981414] env[78392]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 1042.981414] env[78392]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 1042.981414] env[78392]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 1042.981414] env[78392]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 1042.981414] env[78392]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 1042.981414] env[78392]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 1042.981414] env[78392]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 1042.981414] env[78392]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 1042.981414] env[78392]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 1042.981414] env[78392]: ERROR neutron response = self.get(path=path, params=params) [ 1042.982134] env[78392]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 1042.982134] env[78392]: ERROR neutron raise RuntimeError(msg, last_err) [ 1042.982134] env[78392]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1042.982134] env[78392]: ERROR neutron [ 1045.295545] env[78470]: INFO neutron.common.config [-] Logging enabled! [ 1045.295931] env[78470]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 1045.296072] env[78470]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=78470) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 1045.296334] env[78470]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 1045.296416] env[78470]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 1045.297243] env[78470]: WARNING oslo_policy.policy [None req-f0b67d23-a61f-41c6-812a-607e7beea6d0 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1045.297467] env[78470]: WARNING oslo_policy.policy [None req-f0b67d23-a61f-41c6-812a-607e7beea6d0 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1045.298643] env[78470]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-f0b67d23-a61f-41c6-812a-607e7beea6d0 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 1045.312292] env[78470]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f0b67d23-a61f-41c6-812a-607e7beea6d0 None None] Session token - acquiring [ 1045.313049] env[78470]: DEBUG networking_nsxv3.common.locking [None req-f0b67d23-a61f-41c6-812a-607e7beea6d0 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=78470) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1045.313268] env[78470]: DEBUG oslo_concurrency.lockutils [None req-f0b67d23-a61f-41c6-812a-607e7beea6d0 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78470) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1045.313859] env[78470]: DEBUG oslo_concurrency.lockutils [None req-f0b67d23-a61f-41c6-812a-607e7beea6d0 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78470) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1045.313859] env[78470]: DEBUG oslo_concurrency.lockutils [None req-f0b67d23-a61f-41c6-812a-607e7beea6d0 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78470) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1045.322588] env[78470]: DEBUG oslo_concurrency.lockutils [None req-f0b67d23-a61f-41c6-812a-607e7beea6d0 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78470) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1045.322838] env[78470]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f0b67d23-a61f-41c6-812a-607e7beea6d0 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1045.323009] env[78470]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f0b67d23-a61f-41c6-812a-607e7beea6d0 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=78470) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1047.330144] env[78470]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f0b67d23-a61f-41c6-812a-607e7beea6d0 None None] Session token - acquiring [ 1047.330611] env[78470]: DEBUG networking_nsxv3.common.locking [None req-f0b67d23-a61f-41c6-812a-607e7beea6d0 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=78470) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1047.330759] env[78470]: DEBUG oslo_concurrency.lockutils [None req-f0b67d23-a61f-41c6-812a-607e7beea6d0 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78470) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1047.330885] env[78470]: DEBUG oslo_concurrency.lockutils [None req-f0b67d23-a61f-41c6-812a-607e7beea6d0 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78470) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1047.331193] env[78470]: DEBUG oslo_concurrency.lockutils [None req-f0b67d23-a61f-41c6-812a-607e7beea6d0 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78470) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1047.340741] env[78470]: DEBUG oslo_concurrency.lockutils [None req-f0b67d23-a61f-41c6-812a-607e7beea6d0 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78470) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1047.340974] env[78470]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f0b67d23-a61f-41c6-812a-607e7beea6d0 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1047.341117] env[78470]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f0b67d23-a61f-41c6-812a-607e7beea6d0 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=78470) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}}