[ 478.978382] env[62596]: INFO neutron.common.config [-] Logging enabled! [ 478.978871] env[62596]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 478.978871] env[62596]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=62596) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 478.979096] env[62596]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 478.979188] env[62596]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 478.980053] env[62596]: WARNING oslo_policy.policy [None req-86d53759-b308-45ad-a4f5-eb8f854ca51f None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 478.980284] env[62596]: WARNING oslo_policy.policy [None req-86d53759-b308-45ad-a4f5-eb8f854ca51f None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 478.981434] env[62596]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-86d53759-b308-45ad-a4f5-eb8f854ca51f None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 478.995756] env[62596]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-86d53759-b308-45ad-a4f5-eb8f854ca51f None None] Session token - acquiring [ 478.996571] env[62596]: DEBUG networking_nsxv3.common.locking [None req-86d53759-b308-45ad-a4f5-eb8f854ca51f None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62596) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 478.996770] env[62596]: DEBUG oslo_concurrency.lockutils [None req-86d53759-b308-45ad-a4f5-eb8f854ca51f None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62596) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 478.996898] env[62596]: DEBUG oslo_concurrency.lockutils [None req-86d53759-b308-45ad-a4f5-eb8f854ca51f None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62596) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 478.997323] env[62596]: DEBUG oslo_concurrency.lockutils [None req-86d53759-b308-45ad-a4f5-eb8f854ca51f None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62596) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 479.006461] env[62596]: DEBUG oslo_concurrency.lockutils [None req-86d53759-b308-45ad-a4f5-eb8f854ca51f None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62596) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 479.006695] env[62596]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-86d53759-b308-45ad-a4f5-eb8f854ca51f None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 479.006840] env[62596]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-86d53759-b308-45ad-a4f5-eb8f854ca51f None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=62596) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 481.013102] env[62596]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-86d53759-b308-45ad-a4f5-eb8f854ca51f None None] Session token - acquiring [ 481.013455] env[62596]: DEBUG networking_nsxv3.common.locking [None req-86d53759-b308-45ad-a4f5-eb8f854ca51f None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62596) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 481.013638] env[62596]: DEBUG oslo_concurrency.lockutils [None req-86d53759-b308-45ad-a4f5-eb8f854ca51f None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62596) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 481.013887] env[62596]: DEBUG oslo_concurrency.lockutils [None req-86d53759-b308-45ad-a4f5-eb8f854ca51f None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62596) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 481.014190] env[62596]: DEBUG oslo_concurrency.lockutils [None req-86d53759-b308-45ad-a4f5-eb8f854ca51f None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62596) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 481.023390] env[62596]: DEBUG oslo_concurrency.lockutils [None req-86d53759-b308-45ad-a4f5-eb8f854ca51f None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62596) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 481.023663] env[62596]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-86d53759-b308-45ad-a4f5-eb8f854ca51f None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 481.023810] env[62596]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-86d53759-b308-45ad-a4f5-eb8f854ca51f None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=62596) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 483.024980] env[62596]: CRITICAL neutron [None req-86d53759-b308-45ad-a4f5-eb8f854ca51f None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 483.024980] env[62596]: ERROR neutron Traceback (most recent call last): [ 483.024980] env[62596]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 483.024980] env[62596]: ERROR neutron sys.exit(main()) [ 483.024980] env[62596]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 483.024980] env[62596]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 483.024980] env[62596]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 483.024980] env[62596]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 483.024980] env[62596]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 483.024980] env[62596]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 483.024980] env[62596]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 483.024980] env[62596]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 483.024980] env[62596]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 483.024980] env[62596]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 483.024980] env[62596]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 483.024980] env[62596]: ERROR neutron response = self.get(path=path, params=params) [ 483.025775] env[62596]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 483.025775] env[62596]: ERROR neutron raise RuntimeError(msg, last_err) [ 483.025775] env[62596]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 483.025775] env[62596]: ERROR neutron [ 485.413137] env[62629]: INFO neutron.common.config [-] Logging enabled! [ 485.413607] env[62629]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 485.413663] env[62629]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=62629) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 485.413864] env[62629]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 485.413961] env[62629]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 485.414807] env[62629]: WARNING oslo_policy.policy [None req-e71b1a3f-6072-4d13-805c-901b39dc8280 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 485.415032] env[62629]: WARNING oslo_policy.policy [None req-e71b1a3f-6072-4d13-805c-901b39dc8280 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 485.416255] env[62629]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-e71b1a3f-6072-4d13-805c-901b39dc8280 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 485.429781] env[62629]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e71b1a3f-6072-4d13-805c-901b39dc8280 None None] Session token - acquiring [ 485.430526] env[62629]: DEBUG networking_nsxv3.common.locking [None req-e71b1a3f-6072-4d13-805c-901b39dc8280 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62629) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 485.430729] env[62629]: DEBUG oslo_concurrency.lockutils [None req-e71b1a3f-6072-4d13-805c-901b39dc8280 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62629) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 485.430863] env[62629]: DEBUG oslo_concurrency.lockutils [None req-e71b1a3f-6072-4d13-805c-901b39dc8280 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62629) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 485.431212] env[62629]: DEBUG oslo_concurrency.lockutils [None req-e71b1a3f-6072-4d13-805c-901b39dc8280 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62629) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 485.439845] env[62629]: DEBUG oslo_concurrency.lockutils [None req-e71b1a3f-6072-4d13-805c-901b39dc8280 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62629) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 485.440113] env[62629]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e71b1a3f-6072-4d13-805c-901b39dc8280 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 485.440267] env[62629]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e71b1a3f-6072-4d13-805c-901b39dc8280 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=62629) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 487.450612] env[62629]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e71b1a3f-6072-4d13-805c-901b39dc8280 None None] Session token - acquiring [ 487.450612] env[62629]: DEBUG networking_nsxv3.common.locking [None req-e71b1a3f-6072-4d13-805c-901b39dc8280 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62629) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 487.450612] env[62629]: DEBUG oslo_concurrency.lockutils [None req-e71b1a3f-6072-4d13-805c-901b39dc8280 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62629) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 487.451025] env[62629]: DEBUG oslo_concurrency.lockutils [None req-e71b1a3f-6072-4d13-805c-901b39dc8280 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62629) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 487.451025] env[62629]: DEBUG oslo_concurrency.lockutils [None req-e71b1a3f-6072-4d13-805c-901b39dc8280 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62629) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 487.458581] env[62629]: DEBUG oslo_concurrency.lockutils [None req-e71b1a3f-6072-4d13-805c-901b39dc8280 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62629) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 487.459007] env[62629]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e71b1a3f-6072-4d13-805c-901b39dc8280 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 487.459274] env[62629]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e71b1a3f-6072-4d13-805c-901b39dc8280 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=62629) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 489.461096] env[62629]: CRITICAL neutron [None req-e71b1a3f-6072-4d13-805c-901b39dc8280 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 489.461096] env[62629]: ERROR neutron Traceback (most recent call last): [ 489.461096] env[62629]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 489.461096] env[62629]: ERROR neutron sys.exit(main()) [ 489.461096] env[62629]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 489.461096] env[62629]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 489.461096] env[62629]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 489.461096] env[62629]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 489.461096] env[62629]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 489.461096] env[62629]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 489.461096] env[62629]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 489.461096] env[62629]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 489.461096] env[62629]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 489.461096] env[62629]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 489.461096] env[62629]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 489.461096] env[62629]: ERROR neutron response = self.get(path=path, params=params) [ 489.461810] env[62629]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 489.461810] env[62629]: ERROR neutron raise RuntimeError(msg, last_err) [ 489.461810] env[62629]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 489.461810] env[62629]: ERROR neutron [ 491.923934] env[62656]: INFO neutron.common.config [-] Logging enabled! [ 491.924406] env[62656]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 491.924471] env[62656]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=62656) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 491.924667] env[62656]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 491.924787] env[62656]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 491.925613] env[62656]: WARNING oslo_policy.policy [None req-4ed92a8e-fd67-49ad-81d0-361f58979a59 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 491.925834] env[62656]: WARNING oslo_policy.policy [None req-4ed92a8e-fd67-49ad-81d0-361f58979a59 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 491.927019] env[62656]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-4ed92a8e-fd67-49ad-81d0-361f58979a59 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 491.941224] env[62656]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4ed92a8e-fd67-49ad-81d0-361f58979a59 None None] Session token - acquiring [ 491.941982] env[62656]: DEBUG networking_nsxv3.common.locking [None req-4ed92a8e-fd67-49ad-81d0-361f58979a59 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62656) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 491.942190] env[62656]: DEBUG oslo_concurrency.lockutils [None req-4ed92a8e-fd67-49ad-81d0-361f58979a59 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62656) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 491.942319] env[62656]: DEBUG oslo_concurrency.lockutils [None req-4ed92a8e-fd67-49ad-81d0-361f58979a59 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62656) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 491.942673] env[62656]: DEBUG oslo_concurrency.lockutils [None req-4ed92a8e-fd67-49ad-81d0-361f58979a59 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62656) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 491.951514] env[62656]: DEBUG oslo_concurrency.lockutils [None req-4ed92a8e-fd67-49ad-81d0-361f58979a59 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62656) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 491.951759] env[62656]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4ed92a8e-fd67-49ad-81d0-361f58979a59 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 491.951913] env[62656]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4ed92a8e-fd67-49ad-81d0-361f58979a59 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=62656) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 493.958396] env[62656]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4ed92a8e-fd67-49ad-81d0-361f58979a59 None None] Session token - acquiring [ 493.958880] env[62656]: DEBUG networking_nsxv3.common.locking [None req-4ed92a8e-fd67-49ad-81d0-361f58979a59 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62656) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 493.958954] env[62656]: DEBUG oslo_concurrency.lockutils [None req-4ed92a8e-fd67-49ad-81d0-361f58979a59 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62656) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 493.959102] env[62656]: DEBUG oslo_concurrency.lockutils [None req-4ed92a8e-fd67-49ad-81d0-361f58979a59 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62656) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 493.959366] env[62656]: DEBUG oslo_concurrency.lockutils [None req-4ed92a8e-fd67-49ad-81d0-361f58979a59 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62656) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 493.970005] env[62656]: DEBUG oslo_concurrency.lockutils [None req-4ed92a8e-fd67-49ad-81d0-361f58979a59 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62656) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 493.970272] env[62656]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4ed92a8e-fd67-49ad-81d0-361f58979a59 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 493.970415] env[62656]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4ed92a8e-fd67-49ad-81d0-361f58979a59 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=62656) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 495.973333] env[62656]: CRITICAL neutron [None req-4ed92a8e-fd67-49ad-81d0-361f58979a59 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 495.973333] env[62656]: ERROR neutron Traceback (most recent call last): [ 495.973333] env[62656]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 495.973333] env[62656]: ERROR neutron sys.exit(main()) [ 495.973333] env[62656]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 495.973333] env[62656]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 495.973333] env[62656]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 495.973333] env[62656]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 495.973333] env[62656]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 495.973333] env[62656]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 495.973333] env[62656]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 495.973333] env[62656]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 495.973333] env[62656]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 495.973333] env[62656]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 495.973333] env[62656]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 495.973333] env[62656]: ERROR neutron response = self.get(path=path, params=params) [ 495.974149] env[62656]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 495.974149] env[62656]: ERROR neutron raise RuntimeError(msg, last_err) [ 495.974149] env[62656]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 495.974149] env[62656]: ERROR neutron [ 498.404939] env[62743]: INFO neutron.common.config [-] Logging enabled! [ 498.405343] env[62743]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 498.405440] env[62743]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=62743) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 498.405643] env[62743]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 498.405737] env[62743]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 498.406570] env[62743]: WARNING oslo_policy.policy [None req-86505d65-3fca-4e40-9487-6dacaace68ca None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 498.406824] env[62743]: WARNING oslo_policy.policy [None req-86505d65-3fca-4e40-9487-6dacaace68ca None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 498.407984] env[62743]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-86505d65-3fca-4e40-9487-6dacaace68ca None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 498.421839] env[62743]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-86505d65-3fca-4e40-9487-6dacaace68ca None None] Session token - acquiring [ 498.422573] env[62743]: DEBUG networking_nsxv3.common.locking [None req-86505d65-3fca-4e40-9487-6dacaace68ca None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62743) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 498.422840] env[62743]: DEBUG oslo_concurrency.lockutils [None req-86505d65-3fca-4e40-9487-6dacaace68ca None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62743) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 498.422976] env[62743]: DEBUG oslo_concurrency.lockutils [None req-86505d65-3fca-4e40-9487-6dacaace68ca None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62743) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 498.423327] env[62743]: DEBUG oslo_concurrency.lockutils [None req-86505d65-3fca-4e40-9487-6dacaace68ca None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62743) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 498.433375] env[62743]: DEBUG oslo_concurrency.lockutils [None req-86505d65-3fca-4e40-9487-6dacaace68ca None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62743) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 498.433630] env[62743]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-86505d65-3fca-4e40-9487-6dacaace68ca None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 498.433787] env[62743]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-86505d65-3fca-4e40-9487-6dacaace68ca None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=62743) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 500.439940] env[62743]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-86505d65-3fca-4e40-9487-6dacaace68ca None None] Session token - acquiring [ 500.440315] env[62743]: DEBUG networking_nsxv3.common.locking [None req-86505d65-3fca-4e40-9487-6dacaace68ca None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62743) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 500.440498] env[62743]: DEBUG oslo_concurrency.lockutils [None req-86505d65-3fca-4e40-9487-6dacaace68ca None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62743) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 500.440623] env[62743]: DEBUG oslo_concurrency.lockutils [None req-86505d65-3fca-4e40-9487-6dacaace68ca None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62743) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 500.440910] env[62743]: DEBUG oslo_concurrency.lockutils [None req-86505d65-3fca-4e40-9487-6dacaace68ca None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62743) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 500.450058] env[62743]: DEBUG oslo_concurrency.lockutils [None req-86505d65-3fca-4e40-9487-6dacaace68ca None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62743) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 500.450296] env[62743]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-86505d65-3fca-4e40-9487-6dacaace68ca None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 500.450434] env[62743]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-86505d65-3fca-4e40-9487-6dacaace68ca None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=62743) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 502.452628] env[62743]: CRITICAL neutron [None req-86505d65-3fca-4e40-9487-6dacaace68ca None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 502.452628] env[62743]: ERROR neutron Traceback (most recent call last): [ 502.452628] env[62743]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 502.452628] env[62743]: ERROR neutron sys.exit(main()) [ 502.452628] env[62743]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 502.452628] env[62743]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 502.452628] env[62743]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 502.452628] env[62743]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 502.452628] env[62743]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 502.452628] env[62743]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 502.452628] env[62743]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 502.452628] env[62743]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 502.452628] env[62743]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 502.452628] env[62743]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 502.452628] env[62743]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 502.452628] env[62743]: ERROR neutron response = self.get(path=path, params=params) [ 502.453412] env[62743]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 502.453412] env[62743]: ERROR neutron raise RuntimeError(msg, last_err) [ 502.453412] env[62743]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 502.453412] env[62743]: ERROR neutron [ 504.877236] env[62752]: INFO neutron.common.config [-] Logging enabled! [ 504.877643] env[62752]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 504.877745] env[62752]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=62752) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 504.877951] env[62752]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 504.878049] env[62752]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 504.878867] env[62752]: WARNING oslo_policy.policy [None req-8b8a5f06-2043-4ae9-9d60-3e1fb6f98342 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 504.879089] env[62752]: WARNING oslo_policy.policy [None req-8b8a5f06-2043-4ae9-9d60-3e1fb6f98342 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 504.880283] env[62752]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-8b8a5f06-2043-4ae9-9d60-3e1fb6f98342 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 504.894128] env[62752]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8b8a5f06-2043-4ae9-9d60-3e1fb6f98342 None None] Session token - acquiring [ 504.894868] env[62752]: DEBUG networking_nsxv3.common.locking [None req-8b8a5f06-2043-4ae9-9d60-3e1fb6f98342 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62752) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 504.895059] env[62752]: DEBUG oslo_concurrency.lockutils [None req-8b8a5f06-2043-4ae9-9d60-3e1fb6f98342 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62752) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 504.895200] env[62752]: DEBUG oslo_concurrency.lockutils [None req-8b8a5f06-2043-4ae9-9d60-3e1fb6f98342 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62752) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 504.895653] env[62752]: DEBUG oslo_concurrency.lockutils [None req-8b8a5f06-2043-4ae9-9d60-3e1fb6f98342 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62752) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 504.904639] env[62752]: DEBUG oslo_concurrency.lockutils [None req-8b8a5f06-2043-4ae9-9d60-3e1fb6f98342 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62752) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 504.904881] env[62752]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8b8a5f06-2043-4ae9-9d60-3e1fb6f98342 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 504.905029] env[62752]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8b8a5f06-2043-4ae9-9d60-3e1fb6f98342 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=62752) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 506.910998] env[62752]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8b8a5f06-2043-4ae9-9d60-3e1fb6f98342 None None] Session token - acquiring [ 506.911501] env[62752]: DEBUG networking_nsxv3.common.locking [None req-8b8a5f06-2043-4ae9-9d60-3e1fb6f98342 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62752) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 506.911501] env[62752]: DEBUG oslo_concurrency.lockutils [None req-8b8a5f06-2043-4ae9-9d60-3e1fb6f98342 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62752) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 506.911680] env[62752]: DEBUG oslo_concurrency.lockutils [None req-8b8a5f06-2043-4ae9-9d60-3e1fb6f98342 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62752) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 506.911915] env[62752]: DEBUG oslo_concurrency.lockutils [None req-8b8a5f06-2043-4ae9-9d60-3e1fb6f98342 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62752) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 506.920743] env[62752]: DEBUG oslo_concurrency.lockutils [None req-8b8a5f06-2043-4ae9-9d60-3e1fb6f98342 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62752) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 506.921001] env[62752]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8b8a5f06-2043-4ae9-9d60-3e1fb6f98342 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 506.921147] env[62752]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8b8a5f06-2043-4ae9-9d60-3e1fb6f98342 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=62752) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 508.924279] env[62752]: CRITICAL neutron [None req-8b8a5f06-2043-4ae9-9d60-3e1fb6f98342 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 508.924279] env[62752]: ERROR neutron Traceback (most recent call last): [ 508.924279] env[62752]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 508.924279] env[62752]: ERROR neutron sys.exit(main()) [ 508.924279] env[62752]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 508.924279] env[62752]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 508.924279] env[62752]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 508.924279] env[62752]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 508.924279] env[62752]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 508.924279] env[62752]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 508.924279] env[62752]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 508.924279] env[62752]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 508.924279] env[62752]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 508.924279] env[62752]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 508.924279] env[62752]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 508.924279] env[62752]: ERROR neutron response = self.get(path=path, params=params) [ 508.925744] env[62752]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 508.925744] env[62752]: ERROR neutron raise RuntimeError(msg, last_err) [ 508.925744] env[62752]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 508.925744] env[62752]: ERROR neutron [ 511.400469] env[62758]: INFO neutron.common.config [-] Logging enabled! [ 511.400810] env[62758]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 511.400912] env[62758]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=62758) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 511.401148] env[62758]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 511.401243] env[62758]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 511.402056] env[62758]: WARNING oslo_policy.policy [None req-9b0bce2a-9e82-4dda-ad2c-9b589a7644c7 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 511.402274] env[62758]: WARNING oslo_policy.policy [None req-9b0bce2a-9e82-4dda-ad2c-9b589a7644c7 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 511.403445] env[62758]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-9b0bce2a-9e82-4dda-ad2c-9b589a7644c7 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 511.417812] env[62758]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b0bce2a-9e82-4dda-ad2c-9b589a7644c7 None None] Session token - acquiring [ 511.418554] env[62758]: DEBUG networking_nsxv3.common.locking [None req-9b0bce2a-9e82-4dda-ad2c-9b589a7644c7 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62758) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 511.418754] env[62758]: DEBUG oslo_concurrency.lockutils [None req-9b0bce2a-9e82-4dda-ad2c-9b589a7644c7 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62758) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 511.418884] env[62758]: DEBUG oslo_concurrency.lockutils [None req-9b0bce2a-9e82-4dda-ad2c-9b589a7644c7 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62758) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 511.419237] env[62758]: DEBUG oslo_concurrency.lockutils [None req-9b0bce2a-9e82-4dda-ad2c-9b589a7644c7 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62758) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 511.428179] env[62758]: DEBUG oslo_concurrency.lockutils [None req-9b0bce2a-9e82-4dda-ad2c-9b589a7644c7 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62758) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 511.428439] env[62758]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b0bce2a-9e82-4dda-ad2c-9b589a7644c7 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 511.428596] env[62758]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b0bce2a-9e82-4dda-ad2c-9b589a7644c7 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=62758) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 513.435011] env[62758]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b0bce2a-9e82-4dda-ad2c-9b589a7644c7 None None] Session token - acquiring [ 513.435386] env[62758]: DEBUG networking_nsxv3.common.locking [None req-9b0bce2a-9e82-4dda-ad2c-9b589a7644c7 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62758) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 513.435386] env[62758]: DEBUG oslo_concurrency.lockutils [None req-9b0bce2a-9e82-4dda-ad2c-9b589a7644c7 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62758) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 513.435498] env[62758]: DEBUG oslo_concurrency.lockutils [None req-9b0bce2a-9e82-4dda-ad2c-9b589a7644c7 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62758) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 513.435815] env[62758]: DEBUG oslo_concurrency.lockutils [None req-9b0bce2a-9e82-4dda-ad2c-9b589a7644c7 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62758) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 513.444847] env[62758]: DEBUG oslo_concurrency.lockutils [None req-9b0bce2a-9e82-4dda-ad2c-9b589a7644c7 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62758) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 513.445071] env[62758]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b0bce2a-9e82-4dda-ad2c-9b589a7644c7 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 513.445212] env[62758]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b0bce2a-9e82-4dda-ad2c-9b589a7644c7 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=62758) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 515.448121] env[62758]: CRITICAL neutron [None req-9b0bce2a-9e82-4dda-ad2c-9b589a7644c7 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 515.448121] env[62758]: ERROR neutron Traceback (most recent call last): [ 515.448121] env[62758]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 515.448121] env[62758]: ERROR neutron sys.exit(main()) [ 515.448121] env[62758]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 515.448121] env[62758]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 515.448121] env[62758]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 515.448121] env[62758]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 515.448121] env[62758]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 515.448121] env[62758]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 515.448121] env[62758]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 515.448121] env[62758]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 515.448121] env[62758]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 515.448121] env[62758]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 515.448121] env[62758]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 515.448121] env[62758]: ERROR neutron response = self.get(path=path, params=params) [ 515.449016] env[62758]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 515.449016] env[62758]: ERROR neutron raise RuntimeError(msg, last_err) [ 515.449016] env[62758]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 515.449016] env[62758]: ERROR neutron [ 518.193818] env[62977]: INFO neutron.common.config [-] Logging enabled! [ 518.194146] env[62977]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 518.194257] env[62977]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=62977) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 518.194481] env[62977]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 518.194570] env[62977]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 518.195462] env[62977]: WARNING oslo_policy.policy [None req-bff7a0dc-ddeb-4384-a131-a84bc657c0cc None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 518.195682] env[62977]: WARNING oslo_policy.policy [None req-bff7a0dc-ddeb-4384-a131-a84bc657c0cc None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 518.196922] env[62977]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-bff7a0dc-ddeb-4384-a131-a84bc657c0cc None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 518.211654] env[62977]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bff7a0dc-ddeb-4384-a131-a84bc657c0cc None None] Session token - acquiring [ 518.212514] env[62977]: DEBUG networking_nsxv3.common.locking [None req-bff7a0dc-ddeb-4384-a131-a84bc657c0cc None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62977) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 518.212734] env[62977]: DEBUG oslo_concurrency.lockutils [None req-bff7a0dc-ddeb-4384-a131-a84bc657c0cc None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62977) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 518.212864] env[62977]: DEBUG oslo_concurrency.lockutils [None req-bff7a0dc-ddeb-4384-a131-a84bc657c0cc None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62977) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 518.213238] env[62977]: DEBUG oslo_concurrency.lockutils [None req-bff7a0dc-ddeb-4384-a131-a84bc657c0cc None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62977) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 518.222544] env[62977]: DEBUG oslo_concurrency.lockutils [None req-bff7a0dc-ddeb-4384-a131-a84bc657c0cc None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62977) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 518.222810] env[62977]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bff7a0dc-ddeb-4384-a131-a84bc657c0cc None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 518.222967] env[62977]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bff7a0dc-ddeb-4384-a131-a84bc657c0cc None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=62977) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 520.229285] env[62977]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bff7a0dc-ddeb-4384-a131-a84bc657c0cc None None] Session token - acquiring [ 520.229567] env[62977]: DEBUG networking_nsxv3.common.locking [None req-bff7a0dc-ddeb-4384-a131-a84bc657c0cc None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62977) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 520.229825] env[62977]: DEBUG oslo_concurrency.lockutils [None req-bff7a0dc-ddeb-4384-a131-a84bc657c0cc None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62977) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 520.230001] env[62977]: DEBUG oslo_concurrency.lockutils [None req-bff7a0dc-ddeb-4384-a131-a84bc657c0cc None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62977) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 520.230294] env[62977]: DEBUG oslo_concurrency.lockutils [None req-bff7a0dc-ddeb-4384-a131-a84bc657c0cc None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62977) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 520.239653] env[62977]: DEBUG oslo_concurrency.lockutils [None req-bff7a0dc-ddeb-4384-a131-a84bc657c0cc None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62977) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 520.239893] env[62977]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bff7a0dc-ddeb-4384-a131-a84bc657c0cc None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 520.240049] env[62977]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bff7a0dc-ddeb-4384-a131-a84bc657c0cc None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=62977) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 522.242184] env[62977]: CRITICAL neutron [None req-bff7a0dc-ddeb-4384-a131-a84bc657c0cc None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 522.242184] env[62977]: ERROR neutron Traceback (most recent call last): [ 522.242184] env[62977]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 522.242184] env[62977]: ERROR neutron sys.exit(main()) [ 522.242184] env[62977]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 522.242184] env[62977]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 522.242184] env[62977]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 522.242184] env[62977]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 522.242184] env[62977]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 522.242184] env[62977]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 522.242184] env[62977]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 522.242184] env[62977]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 522.242184] env[62977]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 522.242184] env[62977]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 522.242184] env[62977]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 522.242184] env[62977]: ERROR neutron response = self.get(path=path, params=params) [ 522.243199] env[62977]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 522.243199] env[62977]: ERROR neutron raise RuntimeError(msg, last_err) [ 522.243199] env[62977]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 522.243199] env[62977]: ERROR neutron [ 524.865616] env[63030]: INFO neutron.common.config [-] Logging enabled! [ 524.865862] env[63030]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 524.866772] env[63030]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=63030) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 524.866772] env[63030]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 524.866772] env[63030]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 524.867448] env[63030]: WARNING oslo_policy.policy [None req-5e9a7a9e-d7e4-4be0-b790-75614a3385b1 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 524.867638] env[63030]: WARNING oslo_policy.policy [None req-5e9a7a9e-d7e4-4be0-b790-75614a3385b1 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 524.868891] env[63030]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-5e9a7a9e-d7e4-4be0-b790-75614a3385b1 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 524.883903] env[63030]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5e9a7a9e-d7e4-4be0-b790-75614a3385b1 None None] Session token - acquiring [ 524.884834] env[63030]: DEBUG networking_nsxv3.common.locking [None req-5e9a7a9e-d7e4-4be0-b790-75614a3385b1 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=63030) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 524.885071] env[63030]: DEBUG oslo_concurrency.lockutils [None req-5e9a7a9e-d7e4-4be0-b790-75614a3385b1 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63030) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 524.885195] env[63030]: DEBUG oslo_concurrency.lockutils [None req-5e9a7a9e-d7e4-4be0-b790-75614a3385b1 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63030) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 524.885573] env[63030]: DEBUG oslo_concurrency.lockutils [None req-5e9a7a9e-d7e4-4be0-b790-75614a3385b1 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63030) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 524.895136] env[63030]: DEBUG oslo_concurrency.lockutils [None req-5e9a7a9e-d7e4-4be0-b790-75614a3385b1 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63030) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 524.897022] env[63030]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5e9a7a9e-d7e4-4be0-b790-75614a3385b1 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 524.897022] env[63030]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5e9a7a9e-d7e4-4be0-b790-75614a3385b1 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=63030) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 526.901557] env[63030]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5e9a7a9e-d7e4-4be0-b790-75614a3385b1 None None] Session token - acquiring [ 526.901845] env[63030]: DEBUG networking_nsxv3.common.locking [None req-5e9a7a9e-d7e4-4be0-b790-75614a3385b1 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=63030) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 526.902031] env[63030]: DEBUG oslo_concurrency.lockutils [None req-5e9a7a9e-d7e4-4be0-b790-75614a3385b1 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63030) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 526.902169] env[63030]: DEBUG oslo_concurrency.lockutils [None req-5e9a7a9e-d7e4-4be0-b790-75614a3385b1 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63030) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 526.902440] env[63030]: DEBUG oslo_concurrency.lockutils [None req-5e9a7a9e-d7e4-4be0-b790-75614a3385b1 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63030) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 526.917555] env[63030]: DEBUG oslo_concurrency.lockutils [None req-5e9a7a9e-d7e4-4be0-b790-75614a3385b1 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63030) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 526.917827] env[63030]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5e9a7a9e-d7e4-4be0-b790-75614a3385b1 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 526.917970] env[63030]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5e9a7a9e-d7e4-4be0-b790-75614a3385b1 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=63030) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 528.920880] env[63030]: CRITICAL neutron [None req-5e9a7a9e-d7e4-4be0-b790-75614a3385b1 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 528.920880] env[63030]: ERROR neutron Traceback (most recent call last): [ 528.920880] env[63030]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 528.920880] env[63030]: ERROR neutron sys.exit(main()) [ 528.920880] env[63030]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 528.920880] env[63030]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 528.920880] env[63030]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 528.920880] env[63030]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 528.920880] env[63030]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 528.920880] env[63030]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 528.920880] env[63030]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 528.920880] env[63030]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 528.920880] env[63030]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 528.920880] env[63030]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 528.920880] env[63030]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 528.920880] env[63030]: ERROR neutron response = self.get(path=path, params=params) [ 528.921616] env[63030]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 528.921616] env[63030]: ERROR neutron raise RuntimeError(msg, last_err) [ 528.921616] env[63030]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 528.921616] env[63030]: ERROR neutron [ 531.639438] env[63485]: INFO neutron.common.config [-] Logging enabled! [ 531.639725] env[63485]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 531.639996] env[63485]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=63485) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 531.640262] env[63485]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 531.640353] env[63485]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 531.641245] env[63485]: WARNING oslo_policy.policy [None req-4f5e2c6a-ab46-4c9c-8ac2-8d6227704e51 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 531.641471] env[63485]: WARNING oslo_policy.policy [None req-4f5e2c6a-ab46-4c9c-8ac2-8d6227704e51 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 531.642702] env[63485]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-4f5e2c6a-ab46-4c9c-8ac2-8d6227704e51 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 531.657477] env[63485]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4f5e2c6a-ab46-4c9c-8ac2-8d6227704e51 None None] Session token - acquiring [ 531.658383] env[63485]: DEBUG networking_nsxv3.common.locking [None req-4f5e2c6a-ab46-4c9c-8ac2-8d6227704e51 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=63485) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 531.658653] env[63485]: DEBUG oslo_concurrency.lockutils [None req-4f5e2c6a-ab46-4c9c-8ac2-8d6227704e51 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63485) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 531.658808] env[63485]: DEBUG oslo_concurrency.lockutils [None req-4f5e2c6a-ab46-4c9c-8ac2-8d6227704e51 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63485) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 531.659229] env[63485]: DEBUG oslo_concurrency.lockutils [None req-4f5e2c6a-ab46-4c9c-8ac2-8d6227704e51 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63485) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 531.669740] env[63485]: DEBUG oslo_concurrency.lockutils [None req-4f5e2c6a-ab46-4c9c-8ac2-8d6227704e51 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63485) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 531.670049] env[63485]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4f5e2c6a-ab46-4c9c-8ac2-8d6227704e51 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 531.670223] env[63485]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4f5e2c6a-ab46-4c9c-8ac2-8d6227704e51 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=63485) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 533.676279] env[63485]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4f5e2c6a-ab46-4c9c-8ac2-8d6227704e51 None None] Session token - acquiring [ 533.676813] env[63485]: DEBUG networking_nsxv3.common.locking [None req-4f5e2c6a-ab46-4c9c-8ac2-8d6227704e51 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=63485) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 533.676813] env[63485]: DEBUG oslo_concurrency.lockutils [None req-4f5e2c6a-ab46-4c9c-8ac2-8d6227704e51 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63485) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 533.676942] env[63485]: DEBUG oslo_concurrency.lockutils [None req-4f5e2c6a-ab46-4c9c-8ac2-8d6227704e51 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63485) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 533.677186] env[63485]: DEBUG oslo_concurrency.lockutils [None req-4f5e2c6a-ab46-4c9c-8ac2-8d6227704e51 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63485) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 533.686453] env[63485]: DEBUG oslo_concurrency.lockutils [None req-4f5e2c6a-ab46-4c9c-8ac2-8d6227704e51 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63485) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 533.686662] env[63485]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4f5e2c6a-ab46-4c9c-8ac2-8d6227704e51 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 533.686803] env[63485]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4f5e2c6a-ab46-4c9c-8ac2-8d6227704e51 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=63485) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 535.688938] env[63485]: CRITICAL neutron [None req-4f5e2c6a-ab46-4c9c-8ac2-8d6227704e51 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 535.688938] env[63485]: ERROR neutron Traceback (most recent call last): [ 535.688938] env[63485]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 535.688938] env[63485]: ERROR neutron sys.exit(main()) [ 535.688938] env[63485]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 535.688938] env[63485]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 535.688938] env[63485]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 535.688938] env[63485]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 535.688938] env[63485]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 535.688938] env[63485]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 535.688938] env[63485]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 535.688938] env[63485]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 535.688938] env[63485]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 535.688938] env[63485]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 535.688938] env[63485]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 535.688938] env[63485]: ERROR neutron response = self.get(path=path, params=params) [ 535.689583] env[63485]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 535.689583] env[63485]: ERROR neutron raise RuntimeError(msg, last_err) [ 535.689583] env[63485]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 535.689583] env[63485]: ERROR neutron [ 538.846277] env[63989]: INFO neutron.common.config [-] Logging enabled! [ 538.846547] env[63989]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 538.846833] env[63989]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=63989) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 538.847237] env[63989]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 538.847329] env[63989]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 538.848385] env[63989]: WARNING oslo_policy.policy [None req-d593aa90-0a2f-4ef4-ad66-0ff48a392d30 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 538.848623] env[63989]: WARNING oslo_policy.policy [None req-d593aa90-0a2f-4ef4-ad66-0ff48a392d30 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 538.850151] env[63989]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-d593aa90-0a2f-4ef4-ad66-0ff48a392d30 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 538.865824] env[63989]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d593aa90-0a2f-4ef4-ad66-0ff48a392d30 None None] Session token - acquiring [ 538.866874] env[63989]: DEBUG networking_nsxv3.common.locking [None req-d593aa90-0a2f-4ef4-ad66-0ff48a392d30 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=63989) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 538.867117] env[63989]: DEBUG oslo_concurrency.lockutils [None req-d593aa90-0a2f-4ef4-ad66-0ff48a392d30 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63989) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 538.867248] env[63989]: DEBUG oslo_concurrency.lockutils [None req-d593aa90-0a2f-4ef4-ad66-0ff48a392d30 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63989) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 538.867628] env[63989]: DEBUG oslo_concurrency.lockutils [None req-d593aa90-0a2f-4ef4-ad66-0ff48a392d30 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63989) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 538.877826] env[63989]: DEBUG oslo_concurrency.lockutils [None req-d593aa90-0a2f-4ef4-ad66-0ff48a392d30 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63989) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 538.878098] env[63989]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d593aa90-0a2f-4ef4-ad66-0ff48a392d30 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 538.878535] env[63989]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d593aa90-0a2f-4ef4-ad66-0ff48a392d30 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=63989) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 540.891715] env[63989]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d593aa90-0a2f-4ef4-ad66-0ff48a392d30 None None] Session token - acquiring [ 540.891715] env[63989]: DEBUG networking_nsxv3.common.locking [None req-d593aa90-0a2f-4ef4-ad66-0ff48a392d30 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=63989) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 540.891715] env[63989]: DEBUG oslo_concurrency.lockutils [None req-d593aa90-0a2f-4ef4-ad66-0ff48a392d30 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63989) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 540.892311] env[63989]: DEBUG oslo_concurrency.lockutils [None req-d593aa90-0a2f-4ef4-ad66-0ff48a392d30 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63989) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 540.892311] env[63989]: DEBUG oslo_concurrency.lockutils [None req-d593aa90-0a2f-4ef4-ad66-0ff48a392d30 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63989) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 540.898039] env[63989]: DEBUG oslo_concurrency.lockutils [None req-d593aa90-0a2f-4ef4-ad66-0ff48a392d30 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63989) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 540.898307] env[63989]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d593aa90-0a2f-4ef4-ad66-0ff48a392d30 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 540.898445] env[63989]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d593aa90-0a2f-4ef4-ad66-0ff48a392d30 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=63989) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 542.900922] env[63989]: CRITICAL neutron [None req-d593aa90-0a2f-4ef4-ad66-0ff48a392d30 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 542.900922] env[63989]: ERROR neutron Traceback (most recent call last): [ 542.900922] env[63989]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 542.900922] env[63989]: ERROR neutron sys.exit(main()) [ 542.900922] env[63989]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 542.900922] env[63989]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 542.900922] env[63989]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 542.900922] env[63989]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 542.900922] env[63989]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 542.900922] env[63989]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 542.900922] env[63989]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 542.900922] env[63989]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 542.900922] env[63989]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 542.900922] env[63989]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 542.900922] env[63989]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 542.900922] env[63989]: ERROR neutron response = self.get(path=path, params=params) [ 542.901716] env[63989]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 542.901716] env[63989]: ERROR neutron raise RuntimeError(msg, last_err) [ 542.901716] env[63989]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 542.901716] env[63989]: ERROR neutron [ 545.696569] env[64392]: INFO neutron.common.config [-] Logging enabled! [ 545.696840] env[64392]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 545.697111] env[64392]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=64392) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 545.697353] env[64392]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 545.697647] env[64392]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 545.698295] env[64392]: WARNING oslo_policy.policy [None req-ebe2bf6e-8e5c-4578-b571-a140160c20b2 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 545.698498] env[64392]: WARNING oslo_policy.policy [None req-ebe2bf6e-8e5c-4578-b571-a140160c20b2 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 545.699713] env[64392]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-ebe2bf6e-8e5c-4578-b571-a140160c20b2 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 545.714593] env[64392]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ebe2bf6e-8e5c-4578-b571-a140160c20b2 None None] Session token - acquiring [ 545.716446] env[64392]: DEBUG networking_nsxv3.common.locking [None req-ebe2bf6e-8e5c-4578-b571-a140160c20b2 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=64392) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 545.716654] env[64392]: DEBUG oslo_concurrency.lockutils [None req-ebe2bf6e-8e5c-4578-b571-a140160c20b2 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64392) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 545.716804] env[64392]: DEBUG oslo_concurrency.lockutils [None req-ebe2bf6e-8e5c-4578-b571-a140160c20b2 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64392) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 545.717147] env[64392]: DEBUG oslo_concurrency.lockutils [None req-ebe2bf6e-8e5c-4578-b571-a140160c20b2 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64392) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 545.728407] env[64392]: DEBUG oslo_concurrency.lockutils [None req-ebe2bf6e-8e5c-4578-b571-a140160c20b2 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64392) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 545.728667] env[64392]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ebe2bf6e-8e5c-4578-b571-a140160c20b2 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 545.728813] env[64392]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ebe2bf6e-8e5c-4578-b571-a140160c20b2 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=64392) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 547.738080] env[64392]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ebe2bf6e-8e5c-4578-b571-a140160c20b2 None None] Session token - acquiring [ 547.738080] env[64392]: DEBUG networking_nsxv3.common.locking [None req-ebe2bf6e-8e5c-4578-b571-a140160c20b2 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=64392) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 547.738080] env[64392]: DEBUG oslo_concurrency.lockutils [None req-ebe2bf6e-8e5c-4578-b571-a140160c20b2 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64392) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 547.738541] env[64392]: DEBUG oslo_concurrency.lockutils [None req-ebe2bf6e-8e5c-4578-b571-a140160c20b2 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64392) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 547.738541] env[64392]: DEBUG oslo_concurrency.lockutils [None req-ebe2bf6e-8e5c-4578-b571-a140160c20b2 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64392) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 547.748069] env[64392]: DEBUG oslo_concurrency.lockutils [None req-ebe2bf6e-8e5c-4578-b571-a140160c20b2 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64392) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 547.748069] env[64392]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ebe2bf6e-8e5c-4578-b571-a140160c20b2 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 547.748069] env[64392]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ebe2bf6e-8e5c-4578-b571-a140160c20b2 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=64392) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 549.748884] env[64392]: CRITICAL neutron [None req-ebe2bf6e-8e5c-4578-b571-a140160c20b2 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 549.748884] env[64392]: ERROR neutron Traceback (most recent call last): [ 549.748884] env[64392]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 549.748884] env[64392]: ERROR neutron sys.exit(main()) [ 549.748884] env[64392]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 549.748884] env[64392]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 549.748884] env[64392]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 549.748884] env[64392]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 549.748884] env[64392]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 549.748884] env[64392]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 549.748884] env[64392]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 549.748884] env[64392]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 549.748884] env[64392]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 549.748884] env[64392]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 549.748884] env[64392]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 549.748884] env[64392]: ERROR neutron response = self.get(path=path, params=params) [ 549.749703] env[64392]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 549.749703] env[64392]: ERROR neutron raise RuntimeError(msg, last_err) [ 549.749703] env[64392]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 549.749703] env[64392]: ERROR neutron [ 552.718235] env[64712]: INFO neutron.common.config [-] Logging enabled! [ 552.718724] env[64712]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 552.719277] env[64712]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=64712) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 552.719637] env[64712]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 552.719822] env[64712]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 552.720845] env[64712]: WARNING oslo_policy.policy [None req-551e5b49-0db4-46bf-86bd-865480f07b8a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 552.721184] env[64712]: WARNING oslo_policy.policy [None req-551e5b49-0db4-46bf-86bd-865480f07b8a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 552.722548] env[64712]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-551e5b49-0db4-46bf-86bd-865480f07b8a None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 552.744415] env[64712]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-551e5b49-0db4-46bf-86bd-865480f07b8a None None] Session token - acquiring [ 552.745469] env[64712]: DEBUG networking_nsxv3.common.locking [None req-551e5b49-0db4-46bf-86bd-865480f07b8a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=64712) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 552.746161] env[64712]: DEBUG oslo_concurrency.lockutils [None req-551e5b49-0db4-46bf-86bd-865480f07b8a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64712) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 552.746504] env[64712]: DEBUG oslo_concurrency.lockutils [None req-551e5b49-0db4-46bf-86bd-865480f07b8a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64712) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 552.747120] env[64712]: DEBUG oslo_concurrency.lockutils [None req-551e5b49-0db4-46bf-86bd-865480f07b8a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64712) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 552.759528] env[64712]: DEBUG oslo_concurrency.lockutils [None req-551e5b49-0db4-46bf-86bd-865480f07b8a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64712) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 552.760585] env[64712]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-551e5b49-0db4-46bf-86bd-865480f07b8a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 552.760876] env[64712]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-551e5b49-0db4-46bf-86bd-865480f07b8a None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=64712) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 554.767376] env[64712]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-551e5b49-0db4-46bf-86bd-865480f07b8a None None] Session token - acquiring [ 554.767705] env[64712]: DEBUG networking_nsxv3.common.locking [None req-551e5b49-0db4-46bf-86bd-865480f07b8a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=64712) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 554.767914] env[64712]: DEBUG oslo_concurrency.lockutils [None req-551e5b49-0db4-46bf-86bd-865480f07b8a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64712) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 554.768032] env[64712]: DEBUG oslo_concurrency.lockutils [None req-551e5b49-0db4-46bf-86bd-865480f07b8a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64712) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 554.768326] env[64712]: DEBUG oslo_concurrency.lockutils [None req-551e5b49-0db4-46bf-86bd-865480f07b8a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64712) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 554.778176] env[64712]: DEBUG oslo_concurrency.lockutils [None req-551e5b49-0db4-46bf-86bd-865480f07b8a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64712) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 554.778434] env[64712]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-551e5b49-0db4-46bf-86bd-865480f07b8a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 554.778571] env[64712]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-551e5b49-0db4-46bf-86bd-865480f07b8a None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=64712) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 556.780883] env[64712]: CRITICAL neutron [None req-551e5b49-0db4-46bf-86bd-865480f07b8a None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 556.780883] env[64712]: ERROR neutron Traceback (most recent call last): [ 556.780883] env[64712]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 556.780883] env[64712]: ERROR neutron sys.exit(main()) [ 556.780883] env[64712]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 556.780883] env[64712]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 556.780883] env[64712]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 556.780883] env[64712]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 556.780883] env[64712]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 556.780883] env[64712]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 556.780883] env[64712]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 556.780883] env[64712]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 556.780883] env[64712]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 556.780883] env[64712]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 556.780883] env[64712]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 556.780883] env[64712]: ERROR neutron response = self.get(path=path, params=params) [ 556.781625] env[64712]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 556.781625] env[64712]: ERROR neutron raise RuntimeError(msg, last_err) [ 556.781625] env[64712]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 556.781625] env[64712]: ERROR neutron [ 559.636251] env[65335]: INFO neutron.common.config [-] Logging enabled! [ 559.636251] env[65335]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 559.636701] env[65335]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=65335) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 559.636973] env[65335]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 559.637069] env[65335]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 559.638170] env[65335]: WARNING oslo_policy.policy [None req-fae96a24-0cb1-4a69-9bfd-de65d41be1fb None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 559.638256] env[65335]: WARNING oslo_policy.policy [None req-fae96a24-0cb1-4a69-9bfd-de65d41be1fb None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 559.639641] env[65335]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-fae96a24-0cb1-4a69-9bfd-de65d41be1fb None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 559.654616] env[65335]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fae96a24-0cb1-4a69-9bfd-de65d41be1fb None None] Session token - acquiring [ 559.655494] env[65335]: DEBUG networking_nsxv3.common.locking [None req-fae96a24-0cb1-4a69-9bfd-de65d41be1fb None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=65335) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 559.655704] env[65335]: DEBUG oslo_concurrency.lockutils [None req-fae96a24-0cb1-4a69-9bfd-de65d41be1fb None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65335) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 559.655826] env[65335]: DEBUG oslo_concurrency.lockutils [None req-fae96a24-0cb1-4a69-9bfd-de65d41be1fb None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65335) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 559.656224] env[65335]: DEBUG oslo_concurrency.lockutils [None req-fae96a24-0cb1-4a69-9bfd-de65d41be1fb None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65335) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 559.665816] env[65335]: DEBUG oslo_concurrency.lockutils [None req-fae96a24-0cb1-4a69-9bfd-de65d41be1fb None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65335) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 559.666518] env[65335]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fae96a24-0cb1-4a69-9bfd-de65d41be1fb None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 559.667008] env[65335]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fae96a24-0cb1-4a69-9bfd-de65d41be1fb None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=65335) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 561.673191] env[65335]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fae96a24-0cb1-4a69-9bfd-de65d41be1fb None None] Session token - acquiring [ 561.673491] env[65335]: DEBUG networking_nsxv3.common.locking [None req-fae96a24-0cb1-4a69-9bfd-de65d41be1fb None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=65335) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 561.673709] env[65335]: DEBUG oslo_concurrency.lockutils [None req-fae96a24-0cb1-4a69-9bfd-de65d41be1fb None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65335) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 561.673855] env[65335]: DEBUG oslo_concurrency.lockutils [None req-fae96a24-0cb1-4a69-9bfd-de65d41be1fb None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65335) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 561.674161] env[65335]: DEBUG oslo_concurrency.lockutils [None req-fae96a24-0cb1-4a69-9bfd-de65d41be1fb None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65335) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 561.683953] env[65335]: DEBUG oslo_concurrency.lockutils [None req-fae96a24-0cb1-4a69-9bfd-de65d41be1fb None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65335) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 561.684234] env[65335]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fae96a24-0cb1-4a69-9bfd-de65d41be1fb None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 561.684372] env[65335]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fae96a24-0cb1-4a69-9bfd-de65d41be1fb None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=65335) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 563.687689] env[65335]: CRITICAL neutron [None req-fae96a24-0cb1-4a69-9bfd-de65d41be1fb None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 563.687689] env[65335]: ERROR neutron Traceback (most recent call last): [ 563.687689] env[65335]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 563.687689] env[65335]: ERROR neutron sys.exit(main()) [ 563.687689] env[65335]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 563.687689] env[65335]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 563.687689] env[65335]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 563.687689] env[65335]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 563.687689] env[65335]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 563.687689] env[65335]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 563.687689] env[65335]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 563.687689] env[65335]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 563.687689] env[65335]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 563.687689] env[65335]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 563.687689] env[65335]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 563.687689] env[65335]: ERROR neutron response = self.get(path=path, params=params) [ 563.688831] env[65335]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 563.688831] env[65335]: ERROR neutron raise RuntimeError(msg, last_err) [ 563.688831] env[65335]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 563.688831] env[65335]: ERROR neutron [ 566.535601] env[65915]: INFO neutron.common.config [-] Logging enabled! [ 566.535891] env[65915]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 566.536273] env[65915]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=65915) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 566.536888] env[65915]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 566.536888] env[65915]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 566.538044] env[65915]: WARNING oslo_policy.policy [None req-79a0cf7d-6e9a-43c6-87e8-6ffa943a8d90 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 566.538576] env[65915]: WARNING oslo_policy.policy [None req-79a0cf7d-6e9a-43c6-87e8-6ffa943a8d90 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 566.539731] env[65915]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-79a0cf7d-6e9a-43c6-87e8-6ffa943a8d90 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 566.554788] env[65915]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-79a0cf7d-6e9a-43c6-87e8-6ffa943a8d90 None None] Session token - acquiring [ 566.555718] env[65915]: DEBUG networking_nsxv3.common.locking [None req-79a0cf7d-6e9a-43c6-87e8-6ffa943a8d90 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=65915) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 566.556079] env[65915]: DEBUG oslo_concurrency.lockutils [None req-79a0cf7d-6e9a-43c6-87e8-6ffa943a8d90 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65915) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 566.556289] env[65915]: DEBUG oslo_concurrency.lockutils [None req-79a0cf7d-6e9a-43c6-87e8-6ffa943a8d90 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65915) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 566.557412] env[65915]: DEBUG oslo_concurrency.lockutils [None req-79a0cf7d-6e9a-43c6-87e8-6ffa943a8d90 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65915) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 566.567404] env[65915]: DEBUG oslo_concurrency.lockutils [None req-79a0cf7d-6e9a-43c6-87e8-6ffa943a8d90 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65915) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 566.567782] env[65915]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-79a0cf7d-6e9a-43c6-87e8-6ffa943a8d90 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 566.568050] env[65915]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-79a0cf7d-6e9a-43c6-87e8-6ffa943a8d90 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=65915) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 568.577040] env[65915]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-79a0cf7d-6e9a-43c6-87e8-6ffa943a8d90 None None] Session token - acquiring [ 568.577040] env[65915]: DEBUG networking_nsxv3.common.locking [None req-79a0cf7d-6e9a-43c6-87e8-6ffa943a8d90 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=65915) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 568.577040] env[65915]: DEBUG oslo_concurrency.lockutils [None req-79a0cf7d-6e9a-43c6-87e8-6ffa943a8d90 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65915) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 568.577626] env[65915]: DEBUG oslo_concurrency.lockutils [None req-79a0cf7d-6e9a-43c6-87e8-6ffa943a8d90 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65915) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 568.577626] env[65915]: DEBUG oslo_concurrency.lockutils [None req-79a0cf7d-6e9a-43c6-87e8-6ffa943a8d90 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65915) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 568.588939] env[65915]: DEBUG oslo_concurrency.lockutils [None req-79a0cf7d-6e9a-43c6-87e8-6ffa943a8d90 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65915) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 568.588939] env[65915]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-79a0cf7d-6e9a-43c6-87e8-6ffa943a8d90 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 568.588939] env[65915]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-79a0cf7d-6e9a-43c6-87e8-6ffa943a8d90 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=65915) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 570.591769] env[65915]: CRITICAL neutron [None req-79a0cf7d-6e9a-43c6-87e8-6ffa943a8d90 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 570.591769] env[65915]: ERROR neutron Traceback (most recent call last): [ 570.591769] env[65915]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 570.591769] env[65915]: ERROR neutron sys.exit(main()) [ 570.591769] env[65915]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 570.591769] env[65915]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 570.591769] env[65915]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 570.591769] env[65915]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 570.591769] env[65915]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 570.591769] env[65915]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 570.591769] env[65915]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 570.591769] env[65915]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 570.591769] env[65915]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 570.591769] env[65915]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 570.591769] env[65915]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 570.591769] env[65915]: ERROR neutron response = self.get(path=path, params=params) [ 570.592726] env[65915]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 570.592726] env[65915]: ERROR neutron raise RuntimeError(msg, last_err) [ 570.592726] env[65915]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 570.592726] env[65915]: ERROR neutron [ 573.407931] env[66401]: INFO neutron.common.config [-] Logging enabled! [ 573.408199] env[66401]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 573.408498] env[66401]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=66401) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 573.408733] env[66401]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 573.408851] env[66401]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 573.409956] env[66401]: WARNING oslo_policy.policy [None req-6f4bb1f3-7fdc-4eb5-9f0c-4a18e9709933 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 573.410199] env[66401]: WARNING oslo_policy.policy [None req-6f4bb1f3-7fdc-4eb5-9f0c-4a18e9709933 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 573.411472] env[66401]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-6f4bb1f3-7fdc-4eb5-9f0c-4a18e9709933 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 573.427249] env[66401]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6f4bb1f3-7fdc-4eb5-9f0c-4a18e9709933 None None] Session token - acquiring [ 573.428145] env[66401]: DEBUG networking_nsxv3.common.locking [None req-6f4bb1f3-7fdc-4eb5-9f0c-4a18e9709933 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=66401) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 573.428365] env[66401]: DEBUG oslo_concurrency.lockutils [None req-6f4bb1f3-7fdc-4eb5-9f0c-4a18e9709933 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66401) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 573.428496] env[66401]: DEBUG oslo_concurrency.lockutils [None req-6f4bb1f3-7fdc-4eb5-9f0c-4a18e9709933 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66401) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 573.428872] env[66401]: DEBUG oslo_concurrency.lockutils [None req-6f4bb1f3-7fdc-4eb5-9f0c-4a18e9709933 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66401) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 573.441383] env[66401]: DEBUG oslo_concurrency.lockutils [None req-6f4bb1f3-7fdc-4eb5-9f0c-4a18e9709933 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66401) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 573.441383] env[66401]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6f4bb1f3-7fdc-4eb5-9f0c-4a18e9709933 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 573.441383] env[66401]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6f4bb1f3-7fdc-4eb5-9f0c-4a18e9709933 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=66401) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 575.445573] env[66401]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6f4bb1f3-7fdc-4eb5-9f0c-4a18e9709933 None None] Session token - acquiring [ 575.446075] env[66401]: DEBUG networking_nsxv3.common.locking [None req-6f4bb1f3-7fdc-4eb5-9f0c-4a18e9709933 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=66401) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 575.446075] env[66401]: DEBUG oslo_concurrency.lockutils [None req-6f4bb1f3-7fdc-4eb5-9f0c-4a18e9709933 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66401) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 575.446186] env[66401]: DEBUG oslo_concurrency.lockutils [None req-6f4bb1f3-7fdc-4eb5-9f0c-4a18e9709933 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66401) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 575.446463] env[66401]: DEBUG oslo_concurrency.lockutils [None req-6f4bb1f3-7fdc-4eb5-9f0c-4a18e9709933 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66401) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 575.455751] env[66401]: DEBUG oslo_concurrency.lockutils [None req-6f4bb1f3-7fdc-4eb5-9f0c-4a18e9709933 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66401) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 575.456014] env[66401]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6f4bb1f3-7fdc-4eb5-9f0c-4a18e9709933 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 575.456176] env[66401]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6f4bb1f3-7fdc-4eb5-9f0c-4a18e9709933 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=66401) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 577.459123] env[66401]: CRITICAL neutron [None req-6f4bb1f3-7fdc-4eb5-9f0c-4a18e9709933 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 577.459123] env[66401]: ERROR neutron Traceback (most recent call last): [ 577.459123] env[66401]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 577.459123] env[66401]: ERROR neutron sys.exit(main()) [ 577.459123] env[66401]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 577.459123] env[66401]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 577.459123] env[66401]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 577.459123] env[66401]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 577.459123] env[66401]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 577.459123] env[66401]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 577.459123] env[66401]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 577.459123] env[66401]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 577.459123] env[66401]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 577.459123] env[66401]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 577.459123] env[66401]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 577.459123] env[66401]: ERROR neutron response = self.get(path=path, params=params) [ 577.459956] env[66401]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 577.459956] env[66401]: ERROR neutron raise RuntimeError(msg, last_err) [ 577.459956] env[66401]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 577.459956] env[66401]: ERROR neutron [ 580.392658] env[66795]: INFO neutron.common.config [-] Logging enabled! [ 580.392926] env[66795]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 580.393212] env[66795]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=66795) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 580.393473] env[66795]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 580.393563] env[66795]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 580.394437] env[66795]: WARNING oslo_policy.policy [None req-737611d7-c579-48e9-ba59-64c739e9ba2d None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 580.394656] env[66795]: WARNING oslo_policy.policy [None req-737611d7-c579-48e9-ba59-64c739e9ba2d None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 580.396787] env[66795]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-737611d7-c579-48e9-ba59-64c739e9ba2d None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 580.416979] env[66795]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-737611d7-c579-48e9-ba59-64c739e9ba2d None None] Session token - acquiring [ 580.417847] env[66795]: DEBUG networking_nsxv3.common.locking [None req-737611d7-c579-48e9-ba59-64c739e9ba2d None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=66795) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 580.418049] env[66795]: DEBUG oslo_concurrency.lockutils [None req-737611d7-c579-48e9-ba59-64c739e9ba2d None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66795) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 580.418289] env[66795]: DEBUG oslo_concurrency.lockutils [None req-737611d7-c579-48e9-ba59-64c739e9ba2d None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66795) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 580.418535] env[66795]: DEBUG oslo_concurrency.lockutils [None req-737611d7-c579-48e9-ba59-64c739e9ba2d None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66795) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 580.428046] env[66795]: DEBUG oslo_concurrency.lockutils [None req-737611d7-c579-48e9-ba59-64c739e9ba2d None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66795) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 580.428276] env[66795]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-737611d7-c579-48e9-ba59-64c739e9ba2d None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 580.428418] env[66795]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-737611d7-c579-48e9-ba59-64c739e9ba2d None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=66795) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 582.434592] env[66795]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-737611d7-c579-48e9-ba59-64c739e9ba2d None None] Session token - acquiring [ 582.435104] env[66795]: DEBUG networking_nsxv3.common.locking [None req-737611d7-c579-48e9-ba59-64c739e9ba2d None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=66795) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 582.435235] env[66795]: DEBUG oslo_concurrency.lockutils [None req-737611d7-c579-48e9-ba59-64c739e9ba2d None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66795) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 582.435350] env[66795]: DEBUG oslo_concurrency.lockutils [None req-737611d7-c579-48e9-ba59-64c739e9ba2d None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66795) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 582.435633] env[66795]: DEBUG oslo_concurrency.lockutils [None req-737611d7-c579-48e9-ba59-64c739e9ba2d None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66795) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 582.445583] env[66795]: DEBUG oslo_concurrency.lockutils [None req-737611d7-c579-48e9-ba59-64c739e9ba2d None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66795) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 582.446017] env[66795]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-737611d7-c579-48e9-ba59-64c739e9ba2d None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 582.446293] env[66795]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-737611d7-c579-48e9-ba59-64c739e9ba2d None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=66795) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 584.448888] env[66795]: CRITICAL neutron [None req-737611d7-c579-48e9-ba59-64c739e9ba2d None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 584.448888] env[66795]: ERROR neutron Traceback (most recent call last): [ 584.448888] env[66795]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 584.448888] env[66795]: ERROR neutron sys.exit(main()) [ 584.448888] env[66795]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 584.448888] env[66795]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 584.448888] env[66795]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 584.448888] env[66795]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 584.448888] env[66795]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 584.448888] env[66795]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 584.448888] env[66795]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 584.448888] env[66795]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 584.448888] env[66795]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 584.448888] env[66795]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 584.448888] env[66795]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 584.448888] env[66795]: ERROR neutron response = self.get(path=path, params=params) [ 584.450026] env[66795]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 584.450026] env[66795]: ERROR neutron raise RuntimeError(msg, last_err) [ 584.450026] env[66795]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 584.450026] env[66795]: ERROR neutron [ 587.340376] env[67159]: INFO neutron.common.config [-] Logging enabled! [ 587.340376] env[67159]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 587.340376] env[67159]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=67159) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 587.340376] env[67159]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 587.340376] env[67159]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 587.340376] env[67159]: WARNING oslo_policy.policy [None req-c015d79a-6b66-446a-8630-bd4dda38bad5 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 587.340376] env[67159]: WARNING oslo_policy.policy [None req-c015d79a-6b66-446a-8630-bd4dda38bad5 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 587.340376] env[67159]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-c015d79a-6b66-446a-8630-bd4dda38bad5 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 587.355462] env[67159]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c015d79a-6b66-446a-8630-bd4dda38bad5 None None] Session token - acquiring [ 587.356450] env[67159]: DEBUG networking_nsxv3.common.locking [None req-c015d79a-6b66-446a-8630-bd4dda38bad5 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=67159) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 587.356613] env[67159]: DEBUG oslo_concurrency.lockutils [None req-c015d79a-6b66-446a-8630-bd4dda38bad5 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67159) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 587.356750] env[67159]: DEBUG oslo_concurrency.lockutils [None req-c015d79a-6b66-446a-8630-bd4dda38bad5 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67159) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 587.357188] env[67159]: DEBUG oslo_concurrency.lockutils [None req-c015d79a-6b66-446a-8630-bd4dda38bad5 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67159) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 587.368504] env[67159]: DEBUG oslo_concurrency.lockutils [None req-c015d79a-6b66-446a-8630-bd4dda38bad5 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67159) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 587.368733] env[67159]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c015d79a-6b66-446a-8630-bd4dda38bad5 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 587.368900] env[67159]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c015d79a-6b66-446a-8630-bd4dda38bad5 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=67159) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 589.378921] env[67159]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c015d79a-6b66-446a-8630-bd4dda38bad5 None None] Session token - acquiring [ 589.378921] env[67159]: DEBUG networking_nsxv3.common.locking [None req-c015d79a-6b66-446a-8630-bd4dda38bad5 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=67159) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 589.378921] env[67159]: DEBUG oslo_concurrency.lockutils [None req-c015d79a-6b66-446a-8630-bd4dda38bad5 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67159) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 589.379401] env[67159]: DEBUG oslo_concurrency.lockutils [None req-c015d79a-6b66-446a-8630-bd4dda38bad5 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67159) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 589.379401] env[67159]: DEBUG oslo_concurrency.lockutils [None req-c015d79a-6b66-446a-8630-bd4dda38bad5 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67159) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 589.394435] env[67159]: DEBUG oslo_concurrency.lockutils [None req-c015d79a-6b66-446a-8630-bd4dda38bad5 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67159) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 589.394713] env[67159]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c015d79a-6b66-446a-8630-bd4dda38bad5 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 589.394863] env[67159]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c015d79a-6b66-446a-8630-bd4dda38bad5 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=67159) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 591.396871] env[67159]: CRITICAL neutron [None req-c015d79a-6b66-446a-8630-bd4dda38bad5 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 591.396871] env[67159]: ERROR neutron Traceback (most recent call last): [ 591.396871] env[67159]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 591.396871] env[67159]: ERROR neutron sys.exit(main()) [ 591.396871] env[67159]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 591.396871] env[67159]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 591.396871] env[67159]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 591.396871] env[67159]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 591.396871] env[67159]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 591.396871] env[67159]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 591.396871] env[67159]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 591.396871] env[67159]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 591.396871] env[67159]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 591.396871] env[67159]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 591.396871] env[67159]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 591.396871] env[67159]: ERROR neutron response = self.get(path=path, params=params) [ 591.397572] env[67159]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 591.397572] env[67159]: ERROR neutron raise RuntimeError(msg, last_err) [ 591.397572] env[67159]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 591.397572] env[67159]: ERROR neutron [ 594.479799] env[67603]: INFO neutron.common.config [-] Logging enabled! [ 594.480133] env[67603]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 594.480389] env[67603]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=67603) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 594.480713] env[67603]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 594.480713] env[67603]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 594.481569] env[67603]: WARNING oslo_policy.policy [None req-e352fcca-3e3d-432c-9920-36f7514f786e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 594.481820] env[67603]: WARNING oslo_policy.policy [None req-e352fcca-3e3d-432c-9920-36f7514f786e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 594.483053] env[67603]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-e352fcca-3e3d-432c-9920-36f7514f786e None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 594.500335] env[67603]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e352fcca-3e3d-432c-9920-36f7514f786e None None] Session token - acquiring [ 594.501239] env[67603]: DEBUG networking_nsxv3.common.locking [None req-e352fcca-3e3d-432c-9920-36f7514f786e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=67603) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 594.501465] env[67603]: DEBUG oslo_concurrency.lockutils [None req-e352fcca-3e3d-432c-9920-36f7514f786e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67603) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 594.501596] env[67603]: DEBUG oslo_concurrency.lockutils [None req-e352fcca-3e3d-432c-9920-36f7514f786e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67603) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 594.501991] env[67603]: DEBUG oslo_concurrency.lockutils [None req-e352fcca-3e3d-432c-9920-36f7514f786e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67603) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 594.511517] env[67603]: DEBUG oslo_concurrency.lockutils [None req-e352fcca-3e3d-432c-9920-36f7514f786e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67603) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 594.511830] env[67603]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e352fcca-3e3d-432c-9920-36f7514f786e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 594.511990] env[67603]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e352fcca-3e3d-432c-9920-36f7514f786e None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=67603) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 596.518176] env[67603]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e352fcca-3e3d-432c-9920-36f7514f786e None None] Session token - acquiring [ 596.518509] env[67603]: DEBUG networking_nsxv3.common.locking [None req-e352fcca-3e3d-432c-9920-36f7514f786e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=67603) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 596.518708] env[67603]: DEBUG oslo_concurrency.lockutils [None req-e352fcca-3e3d-432c-9920-36f7514f786e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67603) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 596.518836] env[67603]: DEBUG oslo_concurrency.lockutils [None req-e352fcca-3e3d-432c-9920-36f7514f786e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67603) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 596.519110] env[67603]: DEBUG oslo_concurrency.lockutils [None req-e352fcca-3e3d-432c-9920-36f7514f786e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67603) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 596.529167] env[67603]: DEBUG oslo_concurrency.lockutils [None req-e352fcca-3e3d-432c-9920-36f7514f786e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67603) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 596.529496] env[67603]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e352fcca-3e3d-432c-9920-36f7514f786e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 596.531591] env[67603]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e352fcca-3e3d-432c-9920-36f7514f786e None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=67603) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 598.531223] env[67603]: CRITICAL neutron [None req-e352fcca-3e3d-432c-9920-36f7514f786e None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 598.531223] env[67603]: ERROR neutron Traceback (most recent call last): [ 598.531223] env[67603]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 598.531223] env[67603]: ERROR neutron sys.exit(main()) [ 598.531223] env[67603]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 598.531223] env[67603]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 598.531223] env[67603]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 598.531223] env[67603]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 598.531223] env[67603]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 598.531223] env[67603]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 598.531223] env[67603]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 598.531223] env[67603]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 598.531223] env[67603]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 598.531223] env[67603]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 598.531223] env[67603]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 598.531223] env[67603]: ERROR neutron response = self.get(path=path, params=params) [ 598.531984] env[67603]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 598.531984] env[67603]: ERROR neutron raise RuntimeError(msg, last_err) [ 598.531984] env[67603]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 598.531984] env[67603]: ERROR neutron [ 601.514071] env[68056]: INFO neutron.common.config [-] Logging enabled! [ 601.514377] env[68056]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 601.514538] env[68056]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=68056) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 601.514811] env[68056]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 601.514846] env[68056]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 601.515988] env[68056]: WARNING oslo_policy.policy [None req-c5571dfc-a542-40ef-8ab0-de14cce90bc3 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 601.515988] env[68056]: WARNING oslo_policy.policy [None req-c5571dfc-a542-40ef-8ab0-de14cce90bc3 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 601.517165] env[68056]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-c5571dfc-a542-40ef-8ab0-de14cce90bc3 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 601.532627] env[68056]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c5571dfc-a542-40ef-8ab0-de14cce90bc3 None None] Session token - acquiring [ 601.533555] env[68056]: DEBUG networking_nsxv3.common.locking [None req-c5571dfc-a542-40ef-8ab0-de14cce90bc3 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68056) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 601.533914] env[68056]: DEBUG oslo_concurrency.lockutils [None req-c5571dfc-a542-40ef-8ab0-de14cce90bc3 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68056) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 601.534014] env[68056]: DEBUG oslo_concurrency.lockutils [None req-c5571dfc-a542-40ef-8ab0-de14cce90bc3 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68056) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 601.534392] env[68056]: DEBUG oslo_concurrency.lockutils [None req-c5571dfc-a542-40ef-8ab0-de14cce90bc3 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68056) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 601.559876] env[68056]: DEBUG oslo_concurrency.lockutils [None req-c5571dfc-a542-40ef-8ab0-de14cce90bc3 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68056) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 601.560183] env[68056]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c5571dfc-a542-40ef-8ab0-de14cce90bc3 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 601.560346] env[68056]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c5571dfc-a542-40ef-8ab0-de14cce90bc3 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=68056) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 603.567566] env[68056]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c5571dfc-a542-40ef-8ab0-de14cce90bc3 None None] Session token - acquiring [ 603.568068] env[68056]: DEBUG networking_nsxv3.common.locking [None req-c5571dfc-a542-40ef-8ab0-de14cce90bc3 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68056) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 603.568315] env[68056]: DEBUG oslo_concurrency.lockutils [None req-c5571dfc-a542-40ef-8ab0-de14cce90bc3 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68056) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 603.568478] env[68056]: DEBUG oslo_concurrency.lockutils [None req-c5571dfc-a542-40ef-8ab0-de14cce90bc3 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68056) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 603.568790] env[68056]: DEBUG oslo_concurrency.lockutils [None req-c5571dfc-a542-40ef-8ab0-de14cce90bc3 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68056) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 603.579509] env[68056]: DEBUG oslo_concurrency.lockutils [None req-c5571dfc-a542-40ef-8ab0-de14cce90bc3 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68056) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 603.579809] env[68056]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c5571dfc-a542-40ef-8ab0-de14cce90bc3 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 603.580029] env[68056]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c5571dfc-a542-40ef-8ab0-de14cce90bc3 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=68056) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 605.583096] env[68056]: CRITICAL neutron [None req-c5571dfc-a542-40ef-8ab0-de14cce90bc3 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 605.583096] env[68056]: ERROR neutron Traceback (most recent call last): [ 605.583096] env[68056]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 605.583096] env[68056]: ERROR neutron sys.exit(main()) [ 605.583096] env[68056]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 605.583096] env[68056]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 605.583096] env[68056]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 605.583096] env[68056]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 605.583096] env[68056]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 605.583096] env[68056]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 605.583096] env[68056]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 605.583096] env[68056]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 605.583096] env[68056]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 605.583096] env[68056]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 605.583096] env[68056]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 605.583096] env[68056]: ERROR neutron response = self.get(path=path, params=params) [ 605.583819] env[68056]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 605.583819] env[68056]: ERROR neutron raise RuntimeError(msg, last_err) [ 605.583819] env[68056]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 605.583819] env[68056]: ERROR neutron [ 608.335686] env[68582]: INFO neutron.common.config [-] Logging enabled! [ 608.336081] env[68582]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 608.336630] env[68582]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=68582) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 608.336630] env[68582]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 608.336630] env[68582]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 608.337452] env[68582]: WARNING oslo_policy.policy [None req-c3c8743e-91ee-4968-8efd-e0ee7053f2d4 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 608.337675] env[68582]: WARNING oslo_policy.policy [None req-c3c8743e-91ee-4968-8efd-e0ee7053f2d4 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 608.338890] env[68582]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-c3c8743e-91ee-4968-8efd-e0ee7053f2d4 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 608.376767] env[68582]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c3c8743e-91ee-4968-8efd-e0ee7053f2d4 None None] Session token - acquiring [ 608.376767] env[68582]: DEBUG networking_nsxv3.common.locking [None req-c3c8743e-91ee-4968-8efd-e0ee7053f2d4 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68582) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 608.376767] env[68582]: DEBUG oslo_concurrency.lockutils [None req-c3c8743e-91ee-4968-8efd-e0ee7053f2d4 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68582) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 608.378480] env[68582]: DEBUG oslo_concurrency.lockutils [None req-c3c8743e-91ee-4968-8efd-e0ee7053f2d4 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68582) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 608.378480] env[68582]: DEBUG oslo_concurrency.lockutils [None req-c3c8743e-91ee-4968-8efd-e0ee7053f2d4 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68582) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 608.378480] env[68582]: DEBUG oslo_concurrency.lockutils [None req-c3c8743e-91ee-4968-8efd-e0ee7053f2d4 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68582) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 608.378480] env[68582]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c3c8743e-91ee-4968-8efd-e0ee7053f2d4 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 608.378480] env[68582]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c3c8743e-91ee-4968-8efd-e0ee7053f2d4 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=68582) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 610.383950] env[68582]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c3c8743e-91ee-4968-8efd-e0ee7053f2d4 None None] Session token - acquiring [ 610.384266] env[68582]: DEBUG networking_nsxv3.common.locking [None req-c3c8743e-91ee-4968-8efd-e0ee7053f2d4 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68582) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 610.384460] env[68582]: DEBUG oslo_concurrency.lockutils [None req-c3c8743e-91ee-4968-8efd-e0ee7053f2d4 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68582) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 610.384584] env[68582]: DEBUG oslo_concurrency.lockutils [None req-c3c8743e-91ee-4968-8efd-e0ee7053f2d4 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68582) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 610.384864] env[68582]: DEBUG oslo_concurrency.lockutils [None req-c3c8743e-91ee-4968-8efd-e0ee7053f2d4 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68582) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 610.395424] env[68582]: DEBUG oslo_concurrency.lockutils [None req-c3c8743e-91ee-4968-8efd-e0ee7053f2d4 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68582) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 610.395748] env[68582]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c3c8743e-91ee-4968-8efd-e0ee7053f2d4 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 610.395923] env[68582]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c3c8743e-91ee-4968-8efd-e0ee7053f2d4 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=68582) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 612.398984] env[68582]: CRITICAL neutron [None req-c3c8743e-91ee-4968-8efd-e0ee7053f2d4 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 612.398984] env[68582]: ERROR neutron Traceback (most recent call last): [ 612.398984] env[68582]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 612.398984] env[68582]: ERROR neutron sys.exit(main()) [ 612.398984] env[68582]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 612.398984] env[68582]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 612.398984] env[68582]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 612.398984] env[68582]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 612.398984] env[68582]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 612.398984] env[68582]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 612.398984] env[68582]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 612.398984] env[68582]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 612.398984] env[68582]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 612.398984] env[68582]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 612.398984] env[68582]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 612.398984] env[68582]: ERROR neutron response = self.get(path=path, params=params) [ 612.399602] env[68582]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 612.399602] env[68582]: ERROR neutron raise RuntimeError(msg, last_err) [ 612.399602] env[68582]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 612.399602] env[68582]: ERROR neutron [ 615.123441] env[69112]: INFO neutron.common.config [-] Logging enabled! [ 615.123759] env[69112]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 615.124080] env[69112]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=69112) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 615.124332] env[69112]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 615.124450] env[69112]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 615.125392] env[69112]: WARNING oslo_policy.policy [None req-904c770e-2ffc-4856-a16a-8a650e3f9f7c None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 615.125603] env[69112]: WARNING oslo_policy.policy [None req-904c770e-2ffc-4856-a16a-8a650e3f9f7c None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 615.126896] env[69112]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-904c770e-2ffc-4856-a16a-8a650e3f9f7c None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 615.141948] env[69112]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-904c770e-2ffc-4856-a16a-8a650e3f9f7c None None] Session token - acquiring [ 615.142826] env[69112]: DEBUG networking_nsxv3.common.locking [None req-904c770e-2ffc-4856-a16a-8a650e3f9f7c None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69112) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 615.143031] env[69112]: DEBUG oslo_concurrency.lockutils [None req-904c770e-2ffc-4856-a16a-8a650e3f9f7c None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69112) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 615.143893] env[69112]: DEBUG oslo_concurrency.lockutils [None req-904c770e-2ffc-4856-a16a-8a650e3f9f7c None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69112) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 615.143893] env[69112]: DEBUG oslo_concurrency.lockutils [None req-904c770e-2ffc-4856-a16a-8a650e3f9f7c None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69112) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 615.153069] env[69112]: DEBUG oslo_concurrency.lockutils [None req-904c770e-2ffc-4856-a16a-8a650e3f9f7c None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69112) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 615.153343] env[69112]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-904c770e-2ffc-4856-a16a-8a650e3f9f7c None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 615.153492] env[69112]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-904c770e-2ffc-4856-a16a-8a650e3f9f7c None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=69112) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 617.160077] env[69112]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-904c770e-2ffc-4856-a16a-8a650e3f9f7c None None] Session token - acquiring [ 617.160556] env[69112]: DEBUG networking_nsxv3.common.locking [None req-904c770e-2ffc-4856-a16a-8a650e3f9f7c None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69112) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 617.160762] env[69112]: DEBUG oslo_concurrency.lockutils [None req-904c770e-2ffc-4856-a16a-8a650e3f9f7c None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69112) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 617.160869] env[69112]: DEBUG oslo_concurrency.lockutils [None req-904c770e-2ffc-4856-a16a-8a650e3f9f7c None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69112) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 617.161146] env[69112]: DEBUG oslo_concurrency.lockutils [None req-904c770e-2ffc-4856-a16a-8a650e3f9f7c None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69112) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 617.172101] env[69112]: DEBUG oslo_concurrency.lockutils [None req-904c770e-2ffc-4856-a16a-8a650e3f9f7c None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69112) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 617.172101] env[69112]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-904c770e-2ffc-4856-a16a-8a650e3f9f7c None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 617.172101] env[69112]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-904c770e-2ffc-4856-a16a-8a650e3f9f7c None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=69112) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 619.172962] env[69112]: CRITICAL neutron [None req-904c770e-2ffc-4856-a16a-8a650e3f9f7c None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 619.172962] env[69112]: ERROR neutron Traceback (most recent call last): [ 619.172962] env[69112]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 619.172962] env[69112]: ERROR neutron sys.exit(main()) [ 619.172962] env[69112]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 619.172962] env[69112]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 619.172962] env[69112]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 619.172962] env[69112]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 619.172962] env[69112]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 619.172962] env[69112]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 619.172962] env[69112]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 619.172962] env[69112]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 619.172962] env[69112]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 619.172962] env[69112]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 619.172962] env[69112]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 619.172962] env[69112]: ERROR neutron response = self.get(path=path, params=params) [ 619.173698] env[69112]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 619.173698] env[69112]: ERROR neutron raise RuntimeError(msg, last_err) [ 619.173698] env[69112]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 619.173698] env[69112]: ERROR neutron [ 621.922560] env[69531]: INFO neutron.common.config [-] Logging enabled! [ 621.922791] env[69531]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 621.923154] env[69531]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=69531) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 621.923392] env[69531]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 621.923488] env[69531]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 621.924364] env[69531]: WARNING oslo_policy.policy [None req-d0df4329-487b-4bcd-a9a8-966ee2d3fb5f None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 621.924588] env[69531]: WARNING oslo_policy.policy [None req-d0df4329-487b-4bcd-a9a8-966ee2d3fb5f None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 621.925787] env[69531]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-d0df4329-487b-4bcd-a9a8-966ee2d3fb5f None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 621.939680] env[69531]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d0df4329-487b-4bcd-a9a8-966ee2d3fb5f None None] Session token - acquiring [ 621.940520] env[69531]: DEBUG networking_nsxv3.common.locking [None req-d0df4329-487b-4bcd-a9a8-966ee2d3fb5f None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69531) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 621.940779] env[69531]: DEBUG oslo_concurrency.lockutils [None req-d0df4329-487b-4bcd-a9a8-966ee2d3fb5f None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69531) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 621.940921] env[69531]: DEBUG oslo_concurrency.lockutils [None req-d0df4329-487b-4bcd-a9a8-966ee2d3fb5f None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69531) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 621.941912] env[69531]: DEBUG oslo_concurrency.lockutils [None req-d0df4329-487b-4bcd-a9a8-966ee2d3fb5f None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69531) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 621.952314] env[69531]: DEBUG oslo_concurrency.lockutils [None req-d0df4329-487b-4bcd-a9a8-966ee2d3fb5f None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69531) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 621.952609] env[69531]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d0df4329-487b-4bcd-a9a8-966ee2d3fb5f None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 621.952763] env[69531]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d0df4329-487b-4bcd-a9a8-966ee2d3fb5f None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=69531) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 623.959094] env[69531]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d0df4329-487b-4bcd-a9a8-966ee2d3fb5f None None] Session token - acquiring [ 623.959448] env[69531]: DEBUG networking_nsxv3.common.locking [None req-d0df4329-487b-4bcd-a9a8-966ee2d3fb5f None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69531) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 623.959584] env[69531]: DEBUG oslo_concurrency.lockutils [None req-d0df4329-487b-4bcd-a9a8-966ee2d3fb5f None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69531) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 623.959705] env[69531]: DEBUG oslo_concurrency.lockutils [None req-d0df4329-487b-4bcd-a9a8-966ee2d3fb5f None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69531) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 623.960061] env[69531]: DEBUG oslo_concurrency.lockutils [None req-d0df4329-487b-4bcd-a9a8-966ee2d3fb5f None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69531) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 623.979600] env[69531]: DEBUG oslo_concurrency.lockutils [None req-d0df4329-487b-4bcd-a9a8-966ee2d3fb5f None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69531) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 623.979858] env[69531]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d0df4329-487b-4bcd-a9a8-966ee2d3fb5f None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 623.980206] env[69531]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d0df4329-487b-4bcd-a9a8-966ee2d3fb5f None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=69531) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 625.981929] env[69531]: CRITICAL neutron [None req-d0df4329-487b-4bcd-a9a8-966ee2d3fb5f None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 625.981929] env[69531]: ERROR neutron Traceback (most recent call last): [ 625.981929] env[69531]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 625.981929] env[69531]: ERROR neutron sys.exit(main()) [ 625.981929] env[69531]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 625.981929] env[69531]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 625.981929] env[69531]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 625.981929] env[69531]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 625.981929] env[69531]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 625.981929] env[69531]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 625.981929] env[69531]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 625.981929] env[69531]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 625.981929] env[69531]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 625.981929] env[69531]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 625.981929] env[69531]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 625.981929] env[69531]: ERROR neutron response = self.get(path=path, params=params) [ 625.982548] env[69531]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 625.982548] env[69531]: ERROR neutron raise RuntimeError(msg, last_err) [ 625.982548] env[69531]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 625.982548] env[69531]: ERROR neutron [ 628.881619] env[69882]: INFO neutron.common.config [-] Logging enabled! [ 628.881619] env[69882]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 628.881619] env[69882]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=69882) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 628.881619] env[69882]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 628.881619] env[69882]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 628.882430] env[69882]: WARNING oslo_policy.policy [None req-3b092417-4fb5-45f7-8e7a-f0ad598a205d None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 628.882645] env[69882]: WARNING oslo_policy.policy [None req-3b092417-4fb5-45f7-8e7a-f0ad598a205d None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 628.884317] env[69882]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-3b092417-4fb5-45f7-8e7a-f0ad598a205d None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 628.898392] env[69882]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3b092417-4fb5-45f7-8e7a-f0ad598a205d None None] Session token - acquiring [ 628.899209] env[69882]: DEBUG networking_nsxv3.common.locking [None req-3b092417-4fb5-45f7-8e7a-f0ad598a205d None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69882) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 628.899411] env[69882]: DEBUG oslo_concurrency.lockutils [None req-3b092417-4fb5-45f7-8e7a-f0ad598a205d None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69882) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 628.899541] env[69882]: DEBUG oslo_concurrency.lockutils [None req-3b092417-4fb5-45f7-8e7a-f0ad598a205d None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69882) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 628.899909] env[69882]: DEBUG oslo_concurrency.lockutils [None req-3b092417-4fb5-45f7-8e7a-f0ad598a205d None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69882) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 628.909104] env[69882]: DEBUG oslo_concurrency.lockutils [None req-3b092417-4fb5-45f7-8e7a-f0ad598a205d None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69882) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 628.909289] env[69882]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3b092417-4fb5-45f7-8e7a-f0ad598a205d None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 628.909442] env[69882]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3b092417-4fb5-45f7-8e7a-f0ad598a205d None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=69882) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 630.917056] env[69882]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3b092417-4fb5-45f7-8e7a-f0ad598a205d None None] Session token - acquiring [ 630.917408] env[69882]: DEBUG networking_nsxv3.common.locking [None req-3b092417-4fb5-45f7-8e7a-f0ad598a205d None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69882) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 630.919787] env[69882]: DEBUG oslo_concurrency.lockutils [None req-3b092417-4fb5-45f7-8e7a-f0ad598a205d None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69882) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 630.919905] env[69882]: DEBUG oslo_concurrency.lockutils [None req-3b092417-4fb5-45f7-8e7a-f0ad598a205d None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69882) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 630.920211] env[69882]: DEBUG oslo_concurrency.lockutils [None req-3b092417-4fb5-45f7-8e7a-f0ad598a205d None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69882) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 630.942088] env[69882]: DEBUG oslo_concurrency.lockutils [None req-3b092417-4fb5-45f7-8e7a-f0ad598a205d None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69882) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 630.942088] env[69882]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3b092417-4fb5-45f7-8e7a-f0ad598a205d None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 630.942088] env[69882]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3b092417-4fb5-45f7-8e7a-f0ad598a205d None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=69882) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 632.932921] env[69882]: CRITICAL neutron [None req-3b092417-4fb5-45f7-8e7a-f0ad598a205d None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 632.932921] env[69882]: ERROR neutron Traceback (most recent call last): [ 632.932921] env[69882]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 632.932921] env[69882]: ERROR neutron sys.exit(main()) [ 632.932921] env[69882]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 632.932921] env[69882]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 632.932921] env[69882]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 632.932921] env[69882]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 632.932921] env[69882]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 632.932921] env[69882]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 632.932921] env[69882]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 632.932921] env[69882]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 632.932921] env[69882]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 632.932921] env[69882]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 632.932921] env[69882]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 632.932921] env[69882]: ERROR neutron response = self.get(path=path, params=params) [ 632.933689] env[69882]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 632.933689] env[69882]: ERROR neutron raise RuntimeError(msg, last_err) [ 632.933689] env[69882]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 632.933689] env[69882]: ERROR neutron [ 635.640305] env[70255]: INFO neutron.common.config [-] Logging enabled! [ 635.640829] env[70255]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 635.641337] env[70255]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=70255) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 635.641671] env[70255]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 635.641861] env[70255]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 635.642797] env[70255]: WARNING oslo_policy.policy [None req-857b9f13-e93e-4c10-b489-c8f600d82241 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 635.643134] env[70255]: WARNING oslo_policy.policy [None req-857b9f13-e93e-4c10-b489-c8f600d82241 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 635.644513] env[70255]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-857b9f13-e93e-4c10-b489-c8f600d82241 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 635.659994] env[70255]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-857b9f13-e93e-4c10-b489-c8f600d82241 None None] Session token - acquiring [ 635.660949] env[70255]: DEBUG networking_nsxv3.common.locking [None req-857b9f13-e93e-4c10-b489-c8f600d82241 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70255) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 635.661320] env[70255]: DEBUG oslo_concurrency.lockutils [None req-857b9f13-e93e-4c10-b489-c8f600d82241 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70255) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 635.661658] env[70255]: DEBUG oslo_concurrency.lockutils [None req-857b9f13-e93e-4c10-b489-c8f600d82241 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70255) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 635.662122] env[70255]: DEBUG oslo_concurrency.lockutils [None req-857b9f13-e93e-4c10-b489-c8f600d82241 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70255) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 635.672491] env[70255]: DEBUG oslo_concurrency.lockutils [None req-857b9f13-e93e-4c10-b489-c8f600d82241 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70255) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 635.672895] env[70255]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-857b9f13-e93e-4c10-b489-c8f600d82241 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 635.673162] env[70255]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-857b9f13-e93e-4c10-b489-c8f600d82241 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=70255) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 637.679426] env[70255]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-857b9f13-e93e-4c10-b489-c8f600d82241 None None] Session token - acquiring [ 637.679931] env[70255]: DEBUG networking_nsxv3.common.locking [None req-857b9f13-e93e-4c10-b489-c8f600d82241 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70255) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 637.680153] env[70255]: DEBUG oslo_concurrency.lockutils [None req-857b9f13-e93e-4c10-b489-c8f600d82241 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70255) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 637.680283] env[70255]: DEBUG oslo_concurrency.lockutils [None req-857b9f13-e93e-4c10-b489-c8f600d82241 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70255) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 637.680560] env[70255]: DEBUG oslo_concurrency.lockutils [None req-857b9f13-e93e-4c10-b489-c8f600d82241 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70255) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 637.691035] env[70255]: DEBUG oslo_concurrency.lockutils [None req-857b9f13-e93e-4c10-b489-c8f600d82241 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70255) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 637.691217] env[70255]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-857b9f13-e93e-4c10-b489-c8f600d82241 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 637.691355] env[70255]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-857b9f13-e93e-4c10-b489-c8f600d82241 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=70255) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 639.693104] env[70255]: CRITICAL neutron [None req-857b9f13-e93e-4c10-b489-c8f600d82241 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 639.693104] env[70255]: ERROR neutron Traceback (most recent call last): [ 639.693104] env[70255]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 639.693104] env[70255]: ERROR neutron sys.exit(main()) [ 639.693104] env[70255]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 639.693104] env[70255]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 639.693104] env[70255]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 639.693104] env[70255]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 639.693104] env[70255]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 639.693104] env[70255]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 639.693104] env[70255]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 639.693104] env[70255]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 639.693104] env[70255]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 639.693104] env[70255]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 639.693104] env[70255]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 639.693104] env[70255]: ERROR neutron response = self.get(path=path, params=params) [ 639.693772] env[70255]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 639.693772] env[70255]: ERROR neutron raise RuntimeError(msg, last_err) [ 639.693772] env[70255]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 639.693772] env[70255]: ERROR neutron [ 642.436500] env[70592]: INFO neutron.common.config [-] Logging enabled! [ 642.436986] env[70592]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 642.437150] env[70592]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=70592) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 642.437245] env[70592]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 642.437336] env[70592]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 642.438175] env[70592]: WARNING oslo_policy.policy [None req-cce4d348-845e-4ca3-9a8b-4eeb1b543762 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 642.438399] env[70592]: WARNING oslo_policy.policy [None req-cce4d348-845e-4ca3-9a8b-4eeb1b543762 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 642.439583] env[70592]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-cce4d348-845e-4ca3-9a8b-4eeb1b543762 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 642.455411] env[70592]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-cce4d348-845e-4ca3-9a8b-4eeb1b543762 None None] Session token - acquiring [ 642.456272] env[70592]: DEBUG networking_nsxv3.common.locking [None req-cce4d348-845e-4ca3-9a8b-4eeb1b543762 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70592) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 642.456339] env[70592]: DEBUG oslo_concurrency.lockutils [None req-cce4d348-845e-4ca3-9a8b-4eeb1b543762 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70592) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 642.456445] env[70592]: DEBUG oslo_concurrency.lockutils [None req-cce4d348-845e-4ca3-9a8b-4eeb1b543762 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70592) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 642.456809] env[70592]: DEBUG oslo_concurrency.lockutils [None req-cce4d348-845e-4ca3-9a8b-4eeb1b543762 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70592) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 642.466082] env[70592]: DEBUG oslo_concurrency.lockutils [None req-cce4d348-845e-4ca3-9a8b-4eeb1b543762 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70592) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 642.466336] env[70592]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-cce4d348-845e-4ca3-9a8b-4eeb1b543762 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 642.466488] env[70592]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-cce4d348-845e-4ca3-9a8b-4eeb1b543762 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=70592) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 644.472539] env[70592]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-cce4d348-845e-4ca3-9a8b-4eeb1b543762 None None] Session token - acquiring [ 644.473204] env[70592]: DEBUG networking_nsxv3.common.locking [None req-cce4d348-845e-4ca3-9a8b-4eeb1b543762 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70592) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 644.473540] env[70592]: DEBUG oslo_concurrency.lockutils [None req-cce4d348-845e-4ca3-9a8b-4eeb1b543762 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70592) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 644.473795] env[70592]: DEBUG oslo_concurrency.lockutils [None req-cce4d348-845e-4ca3-9a8b-4eeb1b543762 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70592) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 644.474168] env[70592]: DEBUG oslo_concurrency.lockutils [None req-cce4d348-845e-4ca3-9a8b-4eeb1b543762 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70592) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 644.483800] env[70592]: DEBUG oslo_concurrency.lockutils [None req-cce4d348-845e-4ca3-9a8b-4eeb1b543762 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70592) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 644.484179] env[70592]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-cce4d348-845e-4ca3-9a8b-4eeb1b543762 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 644.484424] env[70592]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-cce4d348-845e-4ca3-9a8b-4eeb1b543762 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=70592) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 646.488882] env[70592]: CRITICAL neutron [None req-cce4d348-845e-4ca3-9a8b-4eeb1b543762 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 646.488882] env[70592]: ERROR neutron Traceback (most recent call last): [ 646.488882] env[70592]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 646.488882] env[70592]: ERROR neutron sys.exit(main()) [ 646.488882] env[70592]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 646.488882] env[70592]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 646.488882] env[70592]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 646.488882] env[70592]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 646.488882] env[70592]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 646.488882] env[70592]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 646.488882] env[70592]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 646.488882] env[70592]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 646.488882] env[70592]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 646.488882] env[70592]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 646.488882] env[70592]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 646.488882] env[70592]: ERROR neutron response = self.get(path=path, params=params) [ 646.489686] env[70592]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 646.489686] env[70592]: ERROR neutron raise RuntimeError(msg, last_err) [ 646.489686] env[70592]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 646.489686] env[70592]: ERROR neutron [ 649.484515] env[70775]: INFO neutron.common.config [-] Logging enabled! [ 649.484744] env[70775]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 649.485082] env[70775]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=70775) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 649.485337] env[70775]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 649.485431] env[70775]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 649.486272] env[70775]: WARNING oslo_policy.policy [None req-6e7484fa-b13d-4dec-bd1e-3ff5ebda2329 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 649.486494] env[70775]: WARNING oslo_policy.policy [None req-6e7484fa-b13d-4dec-bd1e-3ff5ebda2329 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 649.487695] env[70775]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-6e7484fa-b13d-4dec-bd1e-3ff5ebda2329 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 649.501311] env[70775]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6e7484fa-b13d-4dec-bd1e-3ff5ebda2329 None None] Session token - acquiring [ 649.502154] env[70775]: DEBUG networking_nsxv3.common.locking [None req-6e7484fa-b13d-4dec-bd1e-3ff5ebda2329 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70775) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 649.502790] env[70775]: DEBUG oslo_concurrency.lockutils [None req-6e7484fa-b13d-4dec-bd1e-3ff5ebda2329 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70775) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 649.502883] env[70775]: DEBUG oslo_concurrency.lockutils [None req-6e7484fa-b13d-4dec-bd1e-3ff5ebda2329 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70775) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 649.503260] env[70775]: DEBUG oslo_concurrency.lockutils [None req-6e7484fa-b13d-4dec-bd1e-3ff5ebda2329 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70775) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 649.513324] env[70775]: DEBUG oslo_concurrency.lockutils [None req-6e7484fa-b13d-4dec-bd1e-3ff5ebda2329 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70775) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 649.513580] env[70775]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6e7484fa-b13d-4dec-bd1e-3ff5ebda2329 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 649.513749] env[70775]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6e7484fa-b13d-4dec-bd1e-3ff5ebda2329 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=70775) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 651.519772] env[70775]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6e7484fa-b13d-4dec-bd1e-3ff5ebda2329 None None] Session token - acquiring [ 651.520081] env[70775]: DEBUG networking_nsxv3.common.locking [None req-6e7484fa-b13d-4dec-bd1e-3ff5ebda2329 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70775) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 651.520278] env[70775]: DEBUG oslo_concurrency.lockutils [None req-6e7484fa-b13d-4dec-bd1e-3ff5ebda2329 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70775) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 651.520404] env[70775]: DEBUG oslo_concurrency.lockutils [None req-6e7484fa-b13d-4dec-bd1e-3ff5ebda2329 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70775) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 651.520686] env[70775]: DEBUG oslo_concurrency.lockutils [None req-6e7484fa-b13d-4dec-bd1e-3ff5ebda2329 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70775) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 651.529829] env[70775]: DEBUG oslo_concurrency.lockutils [None req-6e7484fa-b13d-4dec-bd1e-3ff5ebda2329 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70775) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 651.530119] env[70775]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6e7484fa-b13d-4dec-bd1e-3ff5ebda2329 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 651.530283] env[70775]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6e7484fa-b13d-4dec-bd1e-3ff5ebda2329 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=70775) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 653.532908] env[70775]: CRITICAL neutron [None req-6e7484fa-b13d-4dec-bd1e-3ff5ebda2329 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 653.532908] env[70775]: ERROR neutron Traceback (most recent call last): [ 653.532908] env[70775]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 653.532908] env[70775]: ERROR neutron sys.exit(main()) [ 653.532908] env[70775]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 653.532908] env[70775]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 653.532908] env[70775]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 653.532908] env[70775]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 653.532908] env[70775]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 653.532908] env[70775]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 653.532908] env[70775]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 653.532908] env[70775]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 653.532908] env[70775]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 653.532908] env[70775]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 653.532908] env[70775]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 653.532908] env[70775]: ERROR neutron response = self.get(path=path, params=params) [ 653.533853] env[70775]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 653.533853] env[70775]: ERROR neutron raise RuntimeError(msg, last_err) [ 653.533853] env[70775]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 653.533853] env[70775]: ERROR neutron [ 656.232776] env[70845]: INFO neutron.common.config [-] Logging enabled! [ 656.233122] env[70845]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 656.233301] env[70845]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=70845) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 656.233562] env[70845]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 656.233654] env[70845]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 656.234495] env[70845]: WARNING oslo_policy.policy [None req-dd7b0810-483f-4914-accd-e2746e1df057 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 656.234730] env[70845]: WARNING oslo_policy.policy [None req-dd7b0810-483f-4914-accd-e2746e1df057 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 656.235988] env[70845]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-dd7b0810-483f-4914-accd-e2746e1df057 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 656.250230] env[70845]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-dd7b0810-483f-4914-accd-e2746e1df057 None None] Session token - acquiring [ 656.251015] env[70845]: DEBUG networking_nsxv3.common.locking [None req-dd7b0810-483f-4914-accd-e2746e1df057 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70845) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 656.251225] env[70845]: DEBUG oslo_concurrency.lockutils [None req-dd7b0810-483f-4914-accd-e2746e1df057 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70845) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 656.251351] env[70845]: DEBUG oslo_concurrency.lockutils [None req-dd7b0810-483f-4914-accd-e2746e1df057 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70845) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 656.251724] env[70845]: DEBUG oslo_concurrency.lockutils [None req-dd7b0810-483f-4914-accd-e2746e1df057 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70845) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 656.260934] env[70845]: DEBUG oslo_concurrency.lockutils [None req-dd7b0810-483f-4914-accd-e2746e1df057 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70845) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 656.261099] env[70845]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-dd7b0810-483f-4914-accd-e2746e1df057 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 656.261256] env[70845]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-dd7b0810-483f-4914-accd-e2746e1df057 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=70845) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 658.267284] env[70845]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-dd7b0810-483f-4914-accd-e2746e1df057 None None] Session token - acquiring [ 658.267896] env[70845]: DEBUG networking_nsxv3.common.locking [None req-dd7b0810-483f-4914-accd-e2746e1df057 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70845) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 658.268111] env[70845]: DEBUG oslo_concurrency.lockutils [None req-dd7b0810-483f-4914-accd-e2746e1df057 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70845) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 658.268240] env[70845]: DEBUG oslo_concurrency.lockutils [None req-dd7b0810-483f-4914-accd-e2746e1df057 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70845) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 658.268511] env[70845]: DEBUG oslo_concurrency.lockutils [None req-dd7b0810-483f-4914-accd-e2746e1df057 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70845) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 658.277363] env[70845]: DEBUG oslo_concurrency.lockutils [None req-dd7b0810-483f-4914-accd-e2746e1df057 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70845) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 658.277617] env[70845]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-dd7b0810-483f-4914-accd-e2746e1df057 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 658.277737] env[70845]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-dd7b0810-483f-4914-accd-e2746e1df057 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=70845) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 660.280646] env[70845]: CRITICAL neutron [None req-dd7b0810-483f-4914-accd-e2746e1df057 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 660.280646] env[70845]: ERROR neutron Traceback (most recent call last): [ 660.280646] env[70845]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 660.280646] env[70845]: ERROR neutron sys.exit(main()) [ 660.280646] env[70845]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 660.280646] env[70845]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 660.280646] env[70845]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 660.280646] env[70845]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 660.280646] env[70845]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 660.280646] env[70845]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 660.280646] env[70845]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 660.280646] env[70845]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 660.280646] env[70845]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 660.280646] env[70845]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 660.280646] env[70845]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 660.280646] env[70845]: ERROR neutron response = self.get(path=path, params=params) [ 660.281288] env[70845]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 660.281288] env[70845]: ERROR neutron raise RuntimeError(msg, last_err) [ 660.281288] env[70845]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 660.281288] env[70845]: ERROR neutron [ 662.810501] env[70902]: INFO neutron.common.config [-] Logging enabled! [ 662.811002] env[70902]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 662.811143] env[70902]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=70902) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 662.811420] env[70902]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 662.811861] env[70902]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 662.812434] env[70902]: WARNING oslo_policy.policy [None req-0082a92b-34a5-442d-b77c-f26464fbdecc None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 662.812715] env[70902]: WARNING oslo_policy.policy [None req-0082a92b-34a5-442d-b77c-f26464fbdecc None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 662.813933] env[70902]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-0082a92b-34a5-442d-b77c-f26464fbdecc None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 662.828377] env[70902]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0082a92b-34a5-442d-b77c-f26464fbdecc None None] Session token - acquiring [ 662.829195] env[70902]: DEBUG networking_nsxv3.common.locking [None req-0082a92b-34a5-442d-b77c-f26464fbdecc None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70902) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 662.829483] env[70902]: DEBUG oslo_concurrency.lockutils [None req-0082a92b-34a5-442d-b77c-f26464fbdecc None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70902) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 662.829668] env[70902]: DEBUG oslo_concurrency.lockutils [None req-0082a92b-34a5-442d-b77c-f26464fbdecc None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70902) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 662.830069] env[70902]: DEBUG oslo_concurrency.lockutils [None req-0082a92b-34a5-442d-b77c-f26464fbdecc None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70902) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 662.839150] env[70902]: DEBUG oslo_concurrency.lockutils [None req-0082a92b-34a5-442d-b77c-f26464fbdecc None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70902) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 662.839458] env[70902]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0082a92b-34a5-442d-b77c-f26464fbdecc None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 662.839699] env[70902]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0082a92b-34a5-442d-b77c-f26464fbdecc None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=70902) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 664.845818] env[70902]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0082a92b-34a5-442d-b77c-f26464fbdecc None None] Session token - acquiring [ 664.846124] env[70902]: DEBUG networking_nsxv3.common.locking [None req-0082a92b-34a5-442d-b77c-f26464fbdecc None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70902) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 664.846231] env[70902]: DEBUG oslo_concurrency.lockutils [None req-0082a92b-34a5-442d-b77c-f26464fbdecc None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70902) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 664.846355] env[70902]: DEBUG oslo_concurrency.lockutils [None req-0082a92b-34a5-442d-b77c-f26464fbdecc None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70902) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 664.846635] env[70902]: DEBUG oslo_concurrency.lockutils [None req-0082a92b-34a5-442d-b77c-f26464fbdecc None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70902) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 664.855666] env[70902]: DEBUG oslo_concurrency.lockutils [None req-0082a92b-34a5-442d-b77c-f26464fbdecc None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70902) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 664.855908] env[70902]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0082a92b-34a5-442d-b77c-f26464fbdecc None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 664.856052] env[70902]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0082a92b-34a5-442d-b77c-f26464fbdecc None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=70902) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 666.859129] env[70902]: CRITICAL neutron [None req-0082a92b-34a5-442d-b77c-f26464fbdecc None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 666.859129] env[70902]: ERROR neutron Traceback (most recent call last): [ 666.859129] env[70902]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 666.859129] env[70902]: ERROR neutron sys.exit(main()) [ 666.859129] env[70902]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 666.859129] env[70902]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 666.859129] env[70902]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 666.859129] env[70902]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 666.859129] env[70902]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 666.859129] env[70902]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 666.859129] env[70902]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 666.859129] env[70902]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 666.859129] env[70902]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 666.859129] env[70902]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 666.859129] env[70902]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 666.859129] env[70902]: ERROR neutron response = self.get(path=path, params=params) [ 666.859852] env[70902]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 666.859852] env[70902]: ERROR neutron raise RuntimeError(msg, last_err) [ 666.859852] env[70902]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 666.859852] env[70902]: ERROR neutron [ 669.513368] env[70922]: INFO neutron.common.config [-] Logging enabled! [ 669.513727] env[70922]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 669.513961] env[70922]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=70922) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 669.514188] env[70922]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 669.514276] env[70922]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 669.515152] env[70922]: WARNING oslo_policy.policy [None req-82a777d9-2008-40d0-b04c-6c457cdcc7af None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 669.515364] env[70922]: WARNING oslo_policy.policy [None req-82a777d9-2008-40d0-b04c-6c457cdcc7af None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 669.516753] env[70922]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-82a777d9-2008-40d0-b04c-6c457cdcc7af None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 669.531377] env[70922]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-82a777d9-2008-40d0-b04c-6c457cdcc7af None None] Session token - acquiring [ 669.532327] env[70922]: DEBUG networking_nsxv3.common.locking [None req-82a777d9-2008-40d0-b04c-6c457cdcc7af None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70922) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 669.532689] env[70922]: DEBUG oslo_concurrency.lockutils [None req-82a777d9-2008-40d0-b04c-6c457cdcc7af None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70922) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 669.532929] env[70922]: DEBUG oslo_concurrency.lockutils [None req-82a777d9-2008-40d0-b04c-6c457cdcc7af None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70922) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 669.533393] env[70922]: DEBUG oslo_concurrency.lockutils [None req-82a777d9-2008-40d0-b04c-6c457cdcc7af None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70922) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 669.544641] env[70922]: DEBUG oslo_concurrency.lockutils [None req-82a777d9-2008-40d0-b04c-6c457cdcc7af None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70922) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 669.544951] env[70922]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-82a777d9-2008-40d0-b04c-6c457cdcc7af None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 669.545250] env[70922]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-82a777d9-2008-40d0-b04c-6c457cdcc7af None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=70922) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 671.550982] env[70922]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-82a777d9-2008-40d0-b04c-6c457cdcc7af None None] Session token - acquiring [ 671.551273] env[70922]: DEBUG networking_nsxv3.common.locking [None req-82a777d9-2008-40d0-b04c-6c457cdcc7af None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70922) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 671.551452] env[70922]: DEBUG oslo_concurrency.lockutils [None req-82a777d9-2008-40d0-b04c-6c457cdcc7af None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70922) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 671.551572] env[70922]: DEBUG oslo_concurrency.lockutils [None req-82a777d9-2008-40d0-b04c-6c457cdcc7af None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70922) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 671.551886] env[70922]: DEBUG oslo_concurrency.lockutils [None req-82a777d9-2008-40d0-b04c-6c457cdcc7af None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70922) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 671.563086] env[70922]: DEBUG oslo_concurrency.lockutils [None req-82a777d9-2008-40d0-b04c-6c457cdcc7af None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70922) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 671.563326] env[70922]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-82a777d9-2008-40d0-b04c-6c457cdcc7af None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 671.563463] env[70922]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-82a777d9-2008-40d0-b04c-6c457cdcc7af None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=70922) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 673.564951] env[70922]: CRITICAL neutron [None req-82a777d9-2008-40d0-b04c-6c457cdcc7af None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 673.564951] env[70922]: ERROR neutron Traceback (most recent call last): [ 673.564951] env[70922]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 673.564951] env[70922]: ERROR neutron sys.exit(main()) [ 673.564951] env[70922]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 673.564951] env[70922]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 673.564951] env[70922]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 673.564951] env[70922]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 673.564951] env[70922]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 673.564951] env[70922]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 673.564951] env[70922]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 673.564951] env[70922]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 673.564951] env[70922]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 673.564951] env[70922]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 673.564951] env[70922]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 673.564951] env[70922]: ERROR neutron response = self.get(path=path, params=params) [ 673.565784] env[70922]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 673.565784] env[70922]: ERROR neutron raise RuntimeError(msg, last_err) [ 673.565784] env[70922]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 673.565784] env[70922]: ERROR neutron [ 676.314916] env[71037]: INFO neutron.common.config [-] Logging enabled! [ 676.315246] env[71037]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 676.315471] env[71037]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=71037) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 676.315714] env[71037]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 676.315808] env[71037]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 676.317089] env[71037]: WARNING oslo_policy.policy [None req-3aa6bb61-313a-4a14-8947-3adfc88830d8 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 676.317269] env[71037]: WARNING oslo_policy.policy [None req-3aa6bb61-313a-4a14-8947-3adfc88830d8 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 676.318679] env[71037]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-3aa6bb61-313a-4a14-8947-3adfc88830d8 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 676.335077] env[71037]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3aa6bb61-313a-4a14-8947-3adfc88830d8 None None] Session token - acquiring [ 676.336193] env[71037]: DEBUG networking_nsxv3.common.locking [None req-3aa6bb61-313a-4a14-8947-3adfc88830d8 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71037) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 676.336437] env[71037]: DEBUG oslo_concurrency.lockutils [None req-3aa6bb61-313a-4a14-8947-3adfc88830d8 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71037) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 676.336964] env[71037]: DEBUG oslo_concurrency.lockutils [None req-3aa6bb61-313a-4a14-8947-3adfc88830d8 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71037) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 676.336964] env[71037]: DEBUG oslo_concurrency.lockutils [None req-3aa6bb61-313a-4a14-8947-3adfc88830d8 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71037) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 676.346473] env[71037]: DEBUG oslo_concurrency.lockutils [None req-3aa6bb61-313a-4a14-8947-3adfc88830d8 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71037) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 676.346684] env[71037]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3aa6bb61-313a-4a14-8947-3adfc88830d8 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 676.346833] env[71037]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3aa6bb61-313a-4a14-8947-3adfc88830d8 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=71037) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 678.353175] env[71037]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3aa6bb61-313a-4a14-8947-3adfc88830d8 None None] Session token - acquiring [ 678.353467] env[71037]: DEBUG networking_nsxv3.common.locking [None req-3aa6bb61-313a-4a14-8947-3adfc88830d8 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71037) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 678.353467] env[71037]: DEBUG oslo_concurrency.lockutils [None req-3aa6bb61-313a-4a14-8947-3adfc88830d8 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71037) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 678.353560] env[71037]: DEBUG oslo_concurrency.lockutils [None req-3aa6bb61-313a-4a14-8947-3adfc88830d8 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71037) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 678.353844] env[71037]: DEBUG oslo_concurrency.lockutils [None req-3aa6bb61-313a-4a14-8947-3adfc88830d8 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71037) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 678.363493] env[71037]: DEBUG oslo_concurrency.lockutils [None req-3aa6bb61-313a-4a14-8947-3adfc88830d8 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71037) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 678.363761] env[71037]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3aa6bb61-313a-4a14-8947-3adfc88830d8 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 678.363899] env[71037]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3aa6bb61-313a-4a14-8947-3adfc88830d8 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=71037) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 680.364938] env[71037]: CRITICAL neutron [None req-3aa6bb61-313a-4a14-8947-3adfc88830d8 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 680.364938] env[71037]: ERROR neutron Traceback (most recent call last): [ 680.364938] env[71037]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 680.364938] env[71037]: ERROR neutron sys.exit(main()) [ 680.364938] env[71037]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 680.364938] env[71037]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 680.364938] env[71037]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 680.364938] env[71037]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 680.364938] env[71037]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 680.364938] env[71037]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 680.364938] env[71037]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 680.364938] env[71037]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 680.364938] env[71037]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 680.364938] env[71037]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 680.364938] env[71037]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 680.364938] env[71037]: ERROR neutron response = self.get(path=path, params=params) [ 680.365660] env[71037]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 680.365660] env[71037]: ERROR neutron raise RuntimeError(msg, last_err) [ 680.365660] env[71037]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 680.365660] env[71037]: ERROR neutron [ 683.053315] env[71100]: INFO neutron.common.config [-] Logging enabled! [ 683.053625] env[71100]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 683.053876] env[71100]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=71100) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 683.054117] env[71100]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 683.108626] env[71100]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 683.108626] env[71100]: WARNING oslo_policy.policy [None req-889ff4ee-a736-4cbf-b644-f366fa4589ff None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 683.108626] env[71100]: WARNING oslo_policy.policy [None req-889ff4ee-a736-4cbf-b644-f366fa4589ff None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 683.108626] env[71100]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-889ff4ee-a736-4cbf-b644-f366fa4589ff None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 683.121155] env[71100]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-889ff4ee-a736-4cbf-b644-f366fa4589ff None None] Session token - acquiring [ 683.123007] env[71100]: DEBUG networking_nsxv3.common.locking [None req-889ff4ee-a736-4cbf-b644-f366fa4589ff None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71100) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 683.123125] env[71100]: DEBUG oslo_concurrency.lockutils [None req-889ff4ee-a736-4cbf-b644-f366fa4589ff None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71100) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 683.123227] env[71100]: DEBUG oslo_concurrency.lockutils [None req-889ff4ee-a736-4cbf-b644-f366fa4589ff None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71100) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 683.123639] env[71100]: DEBUG oslo_concurrency.lockutils [None req-889ff4ee-a736-4cbf-b644-f366fa4589ff None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71100) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 683.133755] env[71100]: DEBUG oslo_concurrency.lockutils [None req-889ff4ee-a736-4cbf-b644-f366fa4589ff None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71100) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 683.133755] env[71100]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-889ff4ee-a736-4cbf-b644-f366fa4589ff None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 683.133755] env[71100]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-889ff4ee-a736-4cbf-b644-f366fa4589ff None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=71100) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 685.139892] env[71100]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-889ff4ee-a736-4cbf-b644-f366fa4589ff None None] Session token - acquiring [ 685.140296] env[71100]: DEBUG networking_nsxv3.common.locking [None req-889ff4ee-a736-4cbf-b644-f366fa4589ff None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71100) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 685.140419] env[71100]: DEBUG oslo_concurrency.lockutils [None req-889ff4ee-a736-4cbf-b644-f366fa4589ff None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71100) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 685.140543] env[71100]: DEBUG oslo_concurrency.lockutils [None req-889ff4ee-a736-4cbf-b644-f366fa4589ff None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71100) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 685.140816] env[71100]: DEBUG oslo_concurrency.lockutils [None req-889ff4ee-a736-4cbf-b644-f366fa4589ff None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71100) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 685.149828] env[71100]: DEBUG oslo_concurrency.lockutils [None req-889ff4ee-a736-4cbf-b644-f366fa4589ff None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71100) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 685.150060] env[71100]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-889ff4ee-a736-4cbf-b644-f366fa4589ff None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 685.150201] env[71100]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-889ff4ee-a736-4cbf-b644-f366fa4589ff None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=71100) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 687.152978] env[71100]: CRITICAL neutron [None req-889ff4ee-a736-4cbf-b644-f366fa4589ff None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 687.152978] env[71100]: ERROR neutron Traceback (most recent call last): [ 687.152978] env[71100]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 687.152978] env[71100]: ERROR neutron sys.exit(main()) [ 687.152978] env[71100]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 687.152978] env[71100]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 687.152978] env[71100]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 687.152978] env[71100]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 687.152978] env[71100]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 687.152978] env[71100]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 687.152978] env[71100]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 687.152978] env[71100]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 687.152978] env[71100]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 687.152978] env[71100]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 687.152978] env[71100]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 687.152978] env[71100]: ERROR neutron response = self.get(path=path, params=params) [ 687.153644] env[71100]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 687.153644] env[71100]: ERROR neutron raise RuntimeError(msg, last_err) [ 687.153644] env[71100]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 687.153644] env[71100]: ERROR neutron [ 689.799500] env[71159]: INFO neutron.common.config [-] Logging enabled! [ 689.799778] env[71159]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 689.801116] env[71159]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=71159) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 689.801116] env[71159]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 689.801116] env[71159]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 689.801444] env[71159]: WARNING oslo_policy.policy [None req-bc65ced9-5395-49c2-8734-d9a6fbddff1c None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 689.801699] env[71159]: WARNING oslo_policy.policy [None req-bc65ced9-5395-49c2-8734-d9a6fbddff1c None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 689.802942] env[71159]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-bc65ced9-5395-49c2-8734-d9a6fbddff1c None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 689.825376] env[71159]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bc65ced9-5395-49c2-8734-d9a6fbddff1c None None] Session token - acquiring [ 689.826541] env[71159]: DEBUG networking_nsxv3.common.locking [None req-bc65ced9-5395-49c2-8734-d9a6fbddff1c None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71159) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 689.826787] env[71159]: DEBUG oslo_concurrency.lockutils [None req-bc65ced9-5395-49c2-8734-d9a6fbddff1c None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71159) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 689.826917] env[71159]: DEBUG oslo_concurrency.lockutils [None req-bc65ced9-5395-49c2-8734-d9a6fbddff1c None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71159) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 689.827369] env[71159]: DEBUG oslo_concurrency.lockutils [None req-bc65ced9-5395-49c2-8734-d9a6fbddff1c None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71159) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 689.840879] env[71159]: DEBUG oslo_concurrency.lockutils [None req-bc65ced9-5395-49c2-8734-d9a6fbddff1c None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71159) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 689.841267] env[71159]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bc65ced9-5395-49c2-8734-d9a6fbddff1c None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 689.841324] env[71159]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bc65ced9-5395-49c2-8734-d9a6fbddff1c None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=71159) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 691.850240] env[71159]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bc65ced9-5395-49c2-8734-d9a6fbddff1c None None] Session token - acquiring [ 691.850240] env[71159]: DEBUG networking_nsxv3.common.locking [None req-bc65ced9-5395-49c2-8734-d9a6fbddff1c None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71159) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 691.850240] env[71159]: DEBUG oslo_concurrency.lockutils [None req-bc65ced9-5395-49c2-8734-d9a6fbddff1c None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71159) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 691.850565] env[71159]: DEBUG oslo_concurrency.lockutils [None req-bc65ced9-5395-49c2-8734-d9a6fbddff1c None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71159) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 691.850565] env[71159]: DEBUG oslo_concurrency.lockutils [None req-bc65ced9-5395-49c2-8734-d9a6fbddff1c None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71159) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 691.858937] env[71159]: DEBUG oslo_concurrency.lockutils [None req-bc65ced9-5395-49c2-8734-d9a6fbddff1c None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71159) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 691.858937] env[71159]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bc65ced9-5395-49c2-8734-d9a6fbddff1c None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 691.858937] env[71159]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bc65ced9-5395-49c2-8734-d9a6fbddff1c None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=71159) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 693.860896] env[71159]: CRITICAL neutron [None req-bc65ced9-5395-49c2-8734-d9a6fbddff1c None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 693.860896] env[71159]: ERROR neutron Traceback (most recent call last): [ 693.860896] env[71159]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 693.860896] env[71159]: ERROR neutron sys.exit(main()) [ 693.860896] env[71159]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 693.860896] env[71159]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 693.860896] env[71159]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 693.860896] env[71159]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 693.860896] env[71159]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 693.860896] env[71159]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 693.860896] env[71159]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 693.860896] env[71159]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 693.860896] env[71159]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 693.860896] env[71159]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 693.860896] env[71159]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 693.860896] env[71159]: ERROR neutron response = self.get(path=path, params=params) [ 693.861605] env[71159]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 693.861605] env[71159]: ERROR neutron raise RuntimeError(msg, last_err) [ 693.861605] env[71159]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 693.861605] env[71159]: ERROR neutron [ 696.560699] env[71371]: INFO neutron.common.config [-] Logging enabled! [ 696.561060] env[71371]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 696.561230] env[71371]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=71371) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 696.561452] env[71371]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 696.561541] env[71371]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 696.562416] env[71371]: WARNING oslo_policy.policy [None req-de672606-c400-4758-8be0-e4bf42054152 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 696.562606] env[71371]: WARNING oslo_policy.policy [None req-de672606-c400-4758-8be0-e4bf42054152 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 696.563857] env[71371]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-de672606-c400-4758-8be0-e4bf42054152 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 696.587837] env[71371]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-de672606-c400-4758-8be0-e4bf42054152 None None] Session token - acquiring [ 696.588935] env[71371]: DEBUG networking_nsxv3.common.locking [None req-de672606-c400-4758-8be0-e4bf42054152 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71371) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 696.589320] env[71371]: DEBUG oslo_concurrency.lockutils [None req-de672606-c400-4758-8be0-e4bf42054152 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71371) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 696.589524] env[71371]: DEBUG oslo_concurrency.lockutils [None req-de672606-c400-4758-8be0-e4bf42054152 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71371) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 696.590119] env[71371]: DEBUG oslo_concurrency.lockutils [None req-de672606-c400-4758-8be0-e4bf42054152 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71371) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 696.607673] env[71371]: DEBUG oslo_concurrency.lockutils [None req-de672606-c400-4758-8be0-e4bf42054152 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71371) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 696.608334] env[71371]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-de672606-c400-4758-8be0-e4bf42054152 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 696.608599] env[71371]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-de672606-c400-4758-8be0-e4bf42054152 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=71371) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 698.614716] env[71371]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-de672606-c400-4758-8be0-e4bf42054152 None None] Session token - acquiring [ 698.615076] env[71371]: DEBUG networking_nsxv3.common.locking [None req-de672606-c400-4758-8be0-e4bf42054152 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71371) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 698.615290] env[71371]: DEBUG oslo_concurrency.lockutils [None req-de672606-c400-4758-8be0-e4bf42054152 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71371) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 698.615420] env[71371]: DEBUG oslo_concurrency.lockutils [None req-de672606-c400-4758-8be0-e4bf42054152 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71371) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 698.615718] env[71371]: DEBUG oslo_concurrency.lockutils [None req-de672606-c400-4758-8be0-e4bf42054152 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71371) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 698.625241] env[71371]: DEBUG oslo_concurrency.lockutils [None req-de672606-c400-4758-8be0-e4bf42054152 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71371) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 698.625553] env[71371]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-de672606-c400-4758-8be0-e4bf42054152 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 698.625717] env[71371]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-de672606-c400-4758-8be0-e4bf42054152 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=71371) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 700.628652] env[71371]: CRITICAL neutron [None req-de672606-c400-4758-8be0-e4bf42054152 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 700.628652] env[71371]: ERROR neutron Traceback (most recent call last): [ 700.628652] env[71371]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 700.628652] env[71371]: ERROR neutron sys.exit(main()) [ 700.628652] env[71371]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 700.628652] env[71371]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 700.628652] env[71371]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 700.628652] env[71371]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 700.628652] env[71371]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 700.628652] env[71371]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 700.628652] env[71371]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 700.628652] env[71371]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 700.628652] env[71371]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 700.628652] env[71371]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 700.628652] env[71371]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 700.628652] env[71371]: ERROR neutron response = self.get(path=path, params=params) [ 700.629457] env[71371]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 700.629457] env[71371]: ERROR neutron raise RuntimeError(msg, last_err) [ 700.629457] env[71371]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 700.629457] env[71371]: ERROR neutron [ 703.227954] env[71605]: INFO neutron.common.config [-] Logging enabled! [ 703.228255] env[71605]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 703.228535] env[71605]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=71605) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 703.228755] env[71605]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 703.228852] env[71605]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 703.229675] env[71605]: WARNING oslo_policy.policy [None req-d8f302cb-1a15-42d6-b342-fac2d509e28b None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 703.229900] env[71605]: WARNING oslo_policy.policy [None req-d8f302cb-1a15-42d6-b342-fac2d509e28b None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 703.231121] env[71605]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-d8f302cb-1a15-42d6-b342-fac2d509e28b None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 703.254153] env[71605]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d8f302cb-1a15-42d6-b342-fac2d509e28b None None] Session token - acquiring [ 703.255063] env[71605]: DEBUG networking_nsxv3.common.locking [None req-d8f302cb-1a15-42d6-b342-fac2d509e28b None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71605) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 703.255234] env[71605]: DEBUG oslo_concurrency.lockutils [None req-d8f302cb-1a15-42d6-b342-fac2d509e28b None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71605) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 703.255364] env[71605]: DEBUG oslo_concurrency.lockutils [None req-d8f302cb-1a15-42d6-b342-fac2d509e28b None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71605) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 703.255822] env[71605]: DEBUG oslo_concurrency.lockutils [None req-d8f302cb-1a15-42d6-b342-fac2d509e28b None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71605) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 703.265507] env[71605]: DEBUG oslo_concurrency.lockutils [None req-d8f302cb-1a15-42d6-b342-fac2d509e28b None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71605) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 703.265766] env[71605]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d8f302cb-1a15-42d6-b342-fac2d509e28b None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 703.265921] env[71605]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d8f302cb-1a15-42d6-b342-fac2d509e28b None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=71605) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 705.272033] env[71605]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d8f302cb-1a15-42d6-b342-fac2d509e28b None None] Session token - acquiring [ 705.272348] env[71605]: DEBUG networking_nsxv3.common.locking [None req-d8f302cb-1a15-42d6-b342-fac2d509e28b None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71605) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 705.272535] env[71605]: DEBUG oslo_concurrency.lockutils [None req-d8f302cb-1a15-42d6-b342-fac2d509e28b None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71605) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 705.272662] env[71605]: DEBUG oslo_concurrency.lockutils [None req-d8f302cb-1a15-42d6-b342-fac2d509e28b None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71605) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 705.272949] env[71605]: DEBUG oslo_concurrency.lockutils [None req-d8f302cb-1a15-42d6-b342-fac2d509e28b None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71605) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 705.281953] env[71605]: DEBUG oslo_concurrency.lockutils [None req-d8f302cb-1a15-42d6-b342-fac2d509e28b None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71605) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 705.282196] env[71605]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d8f302cb-1a15-42d6-b342-fac2d509e28b None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 705.282339] env[71605]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d8f302cb-1a15-42d6-b342-fac2d509e28b None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=71605) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 707.285120] env[71605]: CRITICAL neutron [None req-d8f302cb-1a15-42d6-b342-fac2d509e28b None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 707.285120] env[71605]: ERROR neutron Traceback (most recent call last): [ 707.285120] env[71605]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 707.285120] env[71605]: ERROR neutron sys.exit(main()) [ 707.285120] env[71605]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 707.285120] env[71605]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 707.285120] env[71605]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 707.285120] env[71605]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 707.285120] env[71605]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 707.285120] env[71605]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 707.285120] env[71605]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 707.285120] env[71605]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 707.285120] env[71605]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 707.285120] env[71605]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 707.285120] env[71605]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 707.285120] env[71605]: ERROR neutron response = self.get(path=path, params=params) [ 707.286068] env[71605]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 707.286068] env[71605]: ERROR neutron raise RuntimeError(msg, last_err) [ 707.286068] env[71605]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 707.286068] env[71605]: ERROR neutron [ 710.087681] env[71728]: INFO neutron.common.config [-] Logging enabled! [ 710.087681] env[71728]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 710.088072] env[71728]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=71728) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 710.088110] env[71728]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 710.088916] env[71728]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 710.089145] env[71728]: WARNING oslo_policy.policy [None req-d5838628-8f69-4cdc-a00e-9a18790b2fee None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 710.089295] env[71728]: WARNING oslo_policy.policy [None req-d5838628-8f69-4cdc-a00e-9a18790b2fee None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 710.090550] env[71728]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-d5838628-8f69-4cdc-a00e-9a18790b2fee None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 710.105523] env[71728]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d5838628-8f69-4cdc-a00e-9a18790b2fee None None] Session token - acquiring [ 710.105811] env[71728]: DEBUG networking_nsxv3.common.locking [None req-d5838628-8f69-4cdc-a00e-9a18790b2fee None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71728) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 710.106071] env[71728]: DEBUG oslo_concurrency.lockutils [None req-d5838628-8f69-4cdc-a00e-9a18790b2fee None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71728) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 710.106215] env[71728]: DEBUG oslo_concurrency.lockutils [None req-d5838628-8f69-4cdc-a00e-9a18790b2fee None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71728) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 710.106590] env[71728]: DEBUG oslo_concurrency.lockutils [None req-d5838628-8f69-4cdc-a00e-9a18790b2fee None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71728) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 710.119476] env[71728]: DEBUG oslo_concurrency.lockutils [None req-d5838628-8f69-4cdc-a00e-9a18790b2fee None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71728) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 710.119810] env[71728]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d5838628-8f69-4cdc-a00e-9a18790b2fee None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 710.119975] env[71728]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d5838628-8f69-4cdc-a00e-9a18790b2fee None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=71728) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 712.126432] env[71728]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d5838628-8f69-4cdc-a00e-9a18790b2fee None None] Session token - acquiring [ 712.126432] env[71728]: DEBUG networking_nsxv3.common.locking [None req-d5838628-8f69-4cdc-a00e-9a18790b2fee None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71728) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 712.126794] env[71728]: DEBUG oslo_concurrency.lockutils [None req-d5838628-8f69-4cdc-a00e-9a18790b2fee None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71728) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 712.126794] env[71728]: DEBUG oslo_concurrency.lockutils [None req-d5838628-8f69-4cdc-a00e-9a18790b2fee None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71728) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 712.126984] env[71728]: DEBUG oslo_concurrency.lockutils [None req-d5838628-8f69-4cdc-a00e-9a18790b2fee None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71728) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 712.135929] env[71728]: DEBUG oslo_concurrency.lockutils [None req-d5838628-8f69-4cdc-a00e-9a18790b2fee None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71728) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 712.136207] env[71728]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d5838628-8f69-4cdc-a00e-9a18790b2fee None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 712.136348] env[71728]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d5838628-8f69-4cdc-a00e-9a18790b2fee None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=71728) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 714.139250] env[71728]: CRITICAL neutron [None req-d5838628-8f69-4cdc-a00e-9a18790b2fee None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 714.139250] env[71728]: ERROR neutron Traceback (most recent call last): [ 714.139250] env[71728]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 714.139250] env[71728]: ERROR neutron sys.exit(main()) [ 714.139250] env[71728]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 714.139250] env[71728]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 714.139250] env[71728]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 714.139250] env[71728]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 714.139250] env[71728]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 714.139250] env[71728]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 714.139250] env[71728]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 714.139250] env[71728]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 714.139250] env[71728]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 714.139250] env[71728]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 714.139250] env[71728]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 714.139250] env[71728]: ERROR neutron response = self.get(path=path, params=params) [ 714.140073] env[71728]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 714.140073] env[71728]: ERROR neutron raise RuntimeError(msg, last_err) [ 714.140073] env[71728]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 714.140073] env[71728]: ERROR neutron [ 716.714540] env[71847]: INFO neutron.common.config [-] Logging enabled! [ 716.714902] env[71847]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 716.715180] env[71847]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=71847) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 716.715330] env[71847]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 716.715419] env[71847]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 716.716311] env[71847]: WARNING oslo_policy.policy [None req-4a599e07-0101-4fbf-b027-0a2752832cfe None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 716.716530] env[71847]: WARNING oslo_policy.policy [None req-4a599e07-0101-4fbf-b027-0a2752832cfe None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 716.717735] env[71847]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-4a599e07-0101-4fbf-b027-0a2752832cfe None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 716.734313] env[71847]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4a599e07-0101-4fbf-b027-0a2752832cfe None None] Session token - acquiring [ 716.735259] env[71847]: DEBUG networking_nsxv3.common.locking [None req-4a599e07-0101-4fbf-b027-0a2752832cfe None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71847) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 716.735466] env[71847]: DEBUG oslo_concurrency.lockutils [None req-4a599e07-0101-4fbf-b027-0a2752832cfe None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71847) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 716.735590] env[71847]: DEBUG oslo_concurrency.lockutils [None req-4a599e07-0101-4fbf-b027-0a2752832cfe None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71847) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 716.735947] env[71847]: DEBUG oslo_concurrency.lockutils [None req-4a599e07-0101-4fbf-b027-0a2752832cfe None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71847) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 716.744991] env[71847]: DEBUG oslo_concurrency.lockutils [None req-4a599e07-0101-4fbf-b027-0a2752832cfe None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71847) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 716.745291] env[71847]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4a599e07-0101-4fbf-b027-0a2752832cfe None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 716.745446] env[71847]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4a599e07-0101-4fbf-b027-0a2752832cfe None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=71847) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 718.751714] env[71847]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4a599e07-0101-4fbf-b027-0a2752832cfe None None] Session token - acquiring [ 718.754274] env[71847]: DEBUG networking_nsxv3.common.locking [None req-4a599e07-0101-4fbf-b027-0a2752832cfe None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71847) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 718.754274] env[71847]: DEBUG oslo_concurrency.lockutils [None req-4a599e07-0101-4fbf-b027-0a2752832cfe None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71847) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 718.754437] env[71847]: DEBUG oslo_concurrency.lockutils [None req-4a599e07-0101-4fbf-b027-0a2752832cfe None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71847) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 718.754437] env[71847]: DEBUG oslo_concurrency.lockutils [None req-4a599e07-0101-4fbf-b027-0a2752832cfe None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71847) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 718.762799] env[71847]: DEBUG oslo_concurrency.lockutils [None req-4a599e07-0101-4fbf-b027-0a2752832cfe None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71847) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 718.762799] env[71847]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4a599e07-0101-4fbf-b027-0a2752832cfe None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 718.762799] env[71847]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4a599e07-0101-4fbf-b027-0a2752832cfe None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=71847) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 720.765078] env[71847]: CRITICAL neutron [None req-4a599e07-0101-4fbf-b027-0a2752832cfe None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 720.765078] env[71847]: ERROR neutron Traceback (most recent call last): [ 720.765078] env[71847]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 720.765078] env[71847]: ERROR neutron sys.exit(main()) [ 720.765078] env[71847]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 720.765078] env[71847]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 720.765078] env[71847]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 720.765078] env[71847]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 720.765078] env[71847]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 720.765078] env[71847]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 720.765078] env[71847]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 720.765078] env[71847]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 720.765078] env[71847]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 720.765078] env[71847]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 720.765078] env[71847]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 720.765078] env[71847]: ERROR neutron response = self.get(path=path, params=params) [ 720.765794] env[71847]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 720.765794] env[71847]: ERROR neutron raise RuntimeError(msg, last_err) [ 720.765794] env[71847]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 720.765794] env[71847]: ERROR neutron [ 723.293498] env[71956]: INFO neutron.common.config [-] Logging enabled! [ 723.293775] env[71956]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 723.294149] env[71956]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=71956) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 723.294496] env[71956]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 723.294496] env[71956]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 723.298006] env[71956]: WARNING oslo_policy.policy [None req-f4fe263b-795a-4e01-84c9-4dd92ab2b926 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 723.298006] env[71956]: WARNING oslo_policy.policy [None req-f4fe263b-795a-4e01-84c9-4dd92ab2b926 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 723.298006] env[71956]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-f4fe263b-795a-4e01-84c9-4dd92ab2b926 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 723.311964] env[71956]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f4fe263b-795a-4e01-84c9-4dd92ab2b926 None None] Session token - acquiring [ 723.312889] env[71956]: DEBUG networking_nsxv3.common.locking [None req-f4fe263b-795a-4e01-84c9-4dd92ab2b926 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71956) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 723.313072] env[71956]: DEBUG oslo_concurrency.lockutils [None req-f4fe263b-795a-4e01-84c9-4dd92ab2b926 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71956) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 723.313199] env[71956]: DEBUG oslo_concurrency.lockutils [None req-f4fe263b-795a-4e01-84c9-4dd92ab2b926 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71956) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 723.313569] env[71956]: DEBUG oslo_concurrency.lockutils [None req-f4fe263b-795a-4e01-84c9-4dd92ab2b926 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71956) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 723.323170] env[71956]: DEBUG oslo_concurrency.lockutils [None req-f4fe263b-795a-4e01-84c9-4dd92ab2b926 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71956) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 723.323441] env[71956]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f4fe263b-795a-4e01-84c9-4dd92ab2b926 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 723.323594] env[71956]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f4fe263b-795a-4e01-84c9-4dd92ab2b926 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=71956) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 725.329814] env[71956]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f4fe263b-795a-4e01-84c9-4dd92ab2b926 None None] Session token - acquiring [ 725.330238] env[71956]: DEBUG networking_nsxv3.common.locking [None req-f4fe263b-795a-4e01-84c9-4dd92ab2b926 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71956) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 725.330428] env[71956]: DEBUG oslo_concurrency.lockutils [None req-f4fe263b-795a-4e01-84c9-4dd92ab2b926 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71956) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 725.330550] env[71956]: DEBUG oslo_concurrency.lockutils [None req-f4fe263b-795a-4e01-84c9-4dd92ab2b926 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71956) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 725.330826] env[71956]: DEBUG oslo_concurrency.lockutils [None req-f4fe263b-795a-4e01-84c9-4dd92ab2b926 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71956) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 725.340475] env[71956]: DEBUG oslo_concurrency.lockutils [None req-f4fe263b-795a-4e01-84c9-4dd92ab2b926 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71956) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 725.340712] env[71956]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f4fe263b-795a-4e01-84c9-4dd92ab2b926 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 725.340914] env[71956]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f4fe263b-795a-4e01-84c9-4dd92ab2b926 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=71956) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 727.343422] env[71956]: CRITICAL neutron [None req-f4fe263b-795a-4e01-84c9-4dd92ab2b926 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 727.343422] env[71956]: ERROR neutron Traceback (most recent call last): [ 727.343422] env[71956]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 727.343422] env[71956]: ERROR neutron sys.exit(main()) [ 727.343422] env[71956]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 727.343422] env[71956]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 727.343422] env[71956]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 727.343422] env[71956]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 727.343422] env[71956]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 727.343422] env[71956]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 727.343422] env[71956]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 727.343422] env[71956]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 727.343422] env[71956]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 727.343422] env[71956]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 727.343422] env[71956]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 727.343422] env[71956]: ERROR neutron response = self.get(path=path, params=params) [ 727.344163] env[71956]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 727.344163] env[71956]: ERROR neutron raise RuntimeError(msg, last_err) [ 727.344163] env[71956]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 727.344163] env[71956]: ERROR neutron [ 730.039673] env[72184]: INFO neutron.common.config [-] Logging enabled! [ 730.039970] env[72184]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 730.040296] env[72184]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=72184) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 730.040551] env[72184]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 730.040645] env[72184]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 730.041498] env[72184]: WARNING oslo_policy.policy [None req-2edb71ba-17a0-4a8b-b227-17fa08946d37 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 730.041721] env[72184]: WARNING oslo_policy.policy [None req-2edb71ba-17a0-4a8b-b227-17fa08946d37 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 730.042929] env[72184]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-2edb71ba-17a0-4a8b-b227-17fa08946d37 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 730.063251] env[72184]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2edb71ba-17a0-4a8b-b227-17fa08946d37 None None] Session token - acquiring [ 730.063251] env[72184]: DEBUG networking_nsxv3.common.locking [None req-2edb71ba-17a0-4a8b-b227-17fa08946d37 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72184) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 730.063251] env[72184]: DEBUG oslo_concurrency.lockutils [None req-2edb71ba-17a0-4a8b-b227-17fa08946d37 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72184) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 730.063493] env[72184]: DEBUG oslo_concurrency.lockutils [None req-2edb71ba-17a0-4a8b-b227-17fa08946d37 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72184) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 730.063765] env[72184]: DEBUG oslo_concurrency.lockutils [None req-2edb71ba-17a0-4a8b-b227-17fa08946d37 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72184) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 730.073082] env[72184]: DEBUG oslo_concurrency.lockutils [None req-2edb71ba-17a0-4a8b-b227-17fa08946d37 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72184) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 730.073418] env[72184]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2edb71ba-17a0-4a8b-b227-17fa08946d37 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 730.073591] env[72184]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2edb71ba-17a0-4a8b-b227-17fa08946d37 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=72184) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 732.083504] env[72184]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2edb71ba-17a0-4a8b-b227-17fa08946d37 None None] Session token - acquiring [ 732.083504] env[72184]: DEBUG networking_nsxv3.common.locking [None req-2edb71ba-17a0-4a8b-b227-17fa08946d37 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72184) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 732.083504] env[72184]: DEBUG oslo_concurrency.lockutils [None req-2edb71ba-17a0-4a8b-b227-17fa08946d37 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72184) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 732.083928] env[72184]: DEBUG oslo_concurrency.lockutils [None req-2edb71ba-17a0-4a8b-b227-17fa08946d37 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72184) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 732.083928] env[72184]: DEBUG oslo_concurrency.lockutils [None req-2edb71ba-17a0-4a8b-b227-17fa08946d37 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72184) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 732.092761] env[72184]: DEBUG oslo_concurrency.lockutils [None req-2edb71ba-17a0-4a8b-b227-17fa08946d37 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72184) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 732.093011] env[72184]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2edb71ba-17a0-4a8b-b227-17fa08946d37 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 732.093146] env[72184]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2edb71ba-17a0-4a8b-b227-17fa08946d37 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=72184) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 734.096037] env[72184]: CRITICAL neutron [None req-2edb71ba-17a0-4a8b-b227-17fa08946d37 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 734.096037] env[72184]: ERROR neutron Traceback (most recent call last): [ 734.096037] env[72184]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 734.096037] env[72184]: ERROR neutron sys.exit(main()) [ 734.096037] env[72184]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 734.096037] env[72184]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 734.096037] env[72184]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 734.096037] env[72184]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 734.096037] env[72184]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 734.096037] env[72184]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 734.096037] env[72184]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 734.096037] env[72184]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 734.096037] env[72184]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 734.096037] env[72184]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 734.096037] env[72184]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 734.096037] env[72184]: ERROR neutron response = self.get(path=path, params=params) [ 734.096748] env[72184]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 734.096748] env[72184]: ERROR neutron raise RuntimeError(msg, last_err) [ 734.096748] env[72184]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 734.096748] env[72184]: ERROR neutron [ 736.803928] env[72367]: INFO neutron.common.config [-] Logging enabled! [ 736.804220] env[72367]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 736.804554] env[72367]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=72367) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 736.804800] env[72367]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 736.804895] env[72367]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 736.805770] env[72367]: WARNING oslo_policy.policy [None req-737625d1-7bd3-4843-80f6-57f6a7be7b51 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 736.805992] env[72367]: WARNING oslo_policy.policy [None req-737625d1-7bd3-4843-80f6-57f6a7be7b51 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 736.807281] env[72367]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-737625d1-7bd3-4843-80f6-57f6a7be7b51 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 736.821595] env[72367]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-737625d1-7bd3-4843-80f6-57f6a7be7b51 None None] Session token - acquiring [ 736.822468] env[72367]: DEBUG networking_nsxv3.common.locking [None req-737625d1-7bd3-4843-80f6-57f6a7be7b51 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72367) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 736.822671] env[72367]: DEBUG oslo_concurrency.lockutils [None req-737625d1-7bd3-4843-80f6-57f6a7be7b51 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72367) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 736.823060] env[72367]: DEBUG oslo_concurrency.lockutils [None req-737625d1-7bd3-4843-80f6-57f6a7be7b51 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72367) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 736.823174] env[72367]: DEBUG oslo_concurrency.lockutils [None req-737625d1-7bd3-4843-80f6-57f6a7be7b51 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72367) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 736.832778] env[72367]: DEBUG oslo_concurrency.lockutils [None req-737625d1-7bd3-4843-80f6-57f6a7be7b51 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72367) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 736.833061] env[72367]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-737625d1-7bd3-4843-80f6-57f6a7be7b51 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 736.833135] env[72367]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-737625d1-7bd3-4843-80f6-57f6a7be7b51 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=72367) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 738.841278] env[72367]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-737625d1-7bd3-4843-80f6-57f6a7be7b51 None None] Session token - acquiring [ 738.841659] env[72367]: DEBUG networking_nsxv3.common.locking [None req-737625d1-7bd3-4843-80f6-57f6a7be7b51 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72367) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 738.841837] env[72367]: DEBUG oslo_concurrency.lockutils [None req-737625d1-7bd3-4843-80f6-57f6a7be7b51 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72367) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 738.841973] env[72367]: DEBUG oslo_concurrency.lockutils [None req-737625d1-7bd3-4843-80f6-57f6a7be7b51 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72367) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 738.842244] env[72367]: DEBUG oslo_concurrency.lockutils [None req-737625d1-7bd3-4843-80f6-57f6a7be7b51 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72367) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 738.851663] env[72367]: DEBUG oslo_concurrency.lockutils [None req-737625d1-7bd3-4843-80f6-57f6a7be7b51 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72367) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 738.851941] env[72367]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-737625d1-7bd3-4843-80f6-57f6a7be7b51 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 738.852111] env[72367]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-737625d1-7bd3-4843-80f6-57f6a7be7b51 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=72367) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 740.853962] env[72367]: CRITICAL neutron [None req-737625d1-7bd3-4843-80f6-57f6a7be7b51 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 740.853962] env[72367]: ERROR neutron Traceback (most recent call last): [ 740.853962] env[72367]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 740.853962] env[72367]: ERROR neutron sys.exit(main()) [ 740.853962] env[72367]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 740.853962] env[72367]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 740.853962] env[72367]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 740.853962] env[72367]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 740.853962] env[72367]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 740.853962] env[72367]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 740.853962] env[72367]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 740.853962] env[72367]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 740.853962] env[72367]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 740.853962] env[72367]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 740.853962] env[72367]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 740.853962] env[72367]: ERROR neutron response = self.get(path=path, params=params) [ 740.854966] env[72367]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 740.854966] env[72367]: ERROR neutron raise RuntimeError(msg, last_err) [ 740.854966] env[72367]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 740.854966] env[72367]: ERROR neutron [ 743.641143] env[72752]: INFO neutron.common.config [-] Logging enabled! [ 743.641395] env[72752]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 743.641675] env[72752]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=72752) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 743.641898] env[72752]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 743.641993] env[72752]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 743.642835] env[72752]: WARNING oslo_policy.policy [None req-1f6db112-50dc-4c13-951a-97caf6ec868a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 743.643062] env[72752]: WARNING oslo_policy.policy [None req-1f6db112-50dc-4c13-951a-97caf6ec868a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 743.644328] env[72752]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-1f6db112-50dc-4c13-951a-97caf6ec868a None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 743.659508] env[72752]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1f6db112-50dc-4c13-951a-97caf6ec868a None None] Session token - acquiring [ 743.660545] env[72752]: DEBUG networking_nsxv3.common.locking [None req-1f6db112-50dc-4c13-951a-97caf6ec868a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72752) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 743.660913] env[72752]: DEBUG oslo_concurrency.lockutils [None req-1f6db112-50dc-4c13-951a-97caf6ec868a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72752) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 743.661154] env[72752]: DEBUG oslo_concurrency.lockutils [None req-1f6db112-50dc-4c13-951a-97caf6ec868a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72752) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 743.661620] env[72752]: DEBUG oslo_concurrency.lockutils [None req-1f6db112-50dc-4c13-951a-97caf6ec868a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72752) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 743.672242] env[72752]: DEBUG oslo_concurrency.lockutils [None req-1f6db112-50dc-4c13-951a-97caf6ec868a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72752) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 743.672496] env[72752]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1f6db112-50dc-4c13-951a-97caf6ec868a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 743.672645] env[72752]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1f6db112-50dc-4c13-951a-97caf6ec868a None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=72752) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 745.679067] env[72752]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1f6db112-50dc-4c13-951a-97caf6ec868a None None] Session token - acquiring [ 745.679452] env[72752]: DEBUG networking_nsxv3.common.locking [None req-1f6db112-50dc-4c13-951a-97caf6ec868a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72752) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 745.679452] env[72752]: DEBUG oslo_concurrency.lockutils [None req-1f6db112-50dc-4c13-951a-97caf6ec868a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72752) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 745.679626] env[72752]: DEBUG oslo_concurrency.lockutils [None req-1f6db112-50dc-4c13-951a-97caf6ec868a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72752) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 745.679785] env[72752]: DEBUG oslo_concurrency.lockutils [None req-1f6db112-50dc-4c13-951a-97caf6ec868a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72752) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 745.689756] env[72752]: DEBUG oslo_concurrency.lockutils [None req-1f6db112-50dc-4c13-951a-97caf6ec868a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72752) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 745.690108] env[72752]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1f6db112-50dc-4c13-951a-97caf6ec868a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 745.690365] env[72752]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1f6db112-50dc-4c13-951a-97caf6ec868a None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=72752) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 747.692931] env[72752]: CRITICAL neutron [None req-1f6db112-50dc-4c13-951a-97caf6ec868a None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 747.692931] env[72752]: ERROR neutron Traceback (most recent call last): [ 747.692931] env[72752]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 747.692931] env[72752]: ERROR neutron sys.exit(main()) [ 747.692931] env[72752]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 747.692931] env[72752]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 747.692931] env[72752]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 747.692931] env[72752]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 747.692931] env[72752]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 747.692931] env[72752]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 747.692931] env[72752]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 747.692931] env[72752]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 747.692931] env[72752]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 747.692931] env[72752]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 747.692931] env[72752]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 747.692931] env[72752]: ERROR neutron response = self.get(path=path, params=params) [ 747.693762] env[72752]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 747.693762] env[72752]: ERROR neutron raise RuntimeError(msg, last_err) [ 747.693762] env[72752]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 747.693762] env[72752]: ERROR neutron [ 750.357666] env[72944]: INFO neutron.common.config [-] Logging enabled! [ 750.358086] env[72944]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 750.358197] env[72944]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=72944) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 750.358416] env[72944]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 750.358506] env[72944]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 750.359402] env[72944]: WARNING oslo_policy.policy [None req-63f2c236-0123-4702-b2f0-6de9bdcfa45b None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 750.359538] env[72944]: WARNING oslo_policy.policy [None req-63f2c236-0123-4702-b2f0-6de9bdcfa45b None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 750.360758] env[72944]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-63f2c236-0123-4702-b2f0-6de9bdcfa45b None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 750.375001] env[72944]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63f2c236-0123-4702-b2f0-6de9bdcfa45b None None] Session token - acquiring [ 750.375823] env[72944]: DEBUG networking_nsxv3.common.locking [None req-63f2c236-0123-4702-b2f0-6de9bdcfa45b None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72944) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 750.376029] env[72944]: DEBUG oslo_concurrency.lockutils [None req-63f2c236-0123-4702-b2f0-6de9bdcfa45b None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72944) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 750.376155] env[72944]: DEBUG oslo_concurrency.lockutils [None req-63f2c236-0123-4702-b2f0-6de9bdcfa45b None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72944) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 750.376521] env[72944]: DEBUG oslo_concurrency.lockutils [None req-63f2c236-0123-4702-b2f0-6de9bdcfa45b None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72944) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 750.385912] env[72944]: DEBUG oslo_concurrency.lockutils [None req-63f2c236-0123-4702-b2f0-6de9bdcfa45b None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72944) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 750.386180] env[72944]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63f2c236-0123-4702-b2f0-6de9bdcfa45b None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 750.386328] env[72944]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63f2c236-0123-4702-b2f0-6de9bdcfa45b None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=72944) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 752.392309] env[72944]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63f2c236-0123-4702-b2f0-6de9bdcfa45b None None] Session token - acquiring [ 752.392801] env[72944]: DEBUG networking_nsxv3.common.locking [None req-63f2c236-0123-4702-b2f0-6de9bdcfa45b None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72944) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 752.392801] env[72944]: DEBUG oslo_concurrency.lockutils [None req-63f2c236-0123-4702-b2f0-6de9bdcfa45b None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72944) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 752.392999] env[72944]: DEBUG oslo_concurrency.lockutils [None req-63f2c236-0123-4702-b2f0-6de9bdcfa45b None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72944) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 752.393167] env[72944]: DEBUG oslo_concurrency.lockutils [None req-63f2c236-0123-4702-b2f0-6de9bdcfa45b None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72944) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 752.402690] env[72944]: DEBUG oslo_concurrency.lockutils [None req-63f2c236-0123-4702-b2f0-6de9bdcfa45b None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72944) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 752.402939] env[72944]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63f2c236-0123-4702-b2f0-6de9bdcfa45b None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 752.403085] env[72944]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63f2c236-0123-4702-b2f0-6de9bdcfa45b None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=72944) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 754.404939] env[72944]: CRITICAL neutron [None req-63f2c236-0123-4702-b2f0-6de9bdcfa45b None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 754.404939] env[72944]: ERROR neutron Traceback (most recent call last): [ 754.404939] env[72944]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 754.404939] env[72944]: ERROR neutron sys.exit(main()) [ 754.404939] env[72944]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 754.404939] env[72944]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 754.404939] env[72944]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 754.404939] env[72944]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 754.404939] env[72944]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 754.404939] env[72944]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 754.404939] env[72944]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 754.404939] env[72944]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 754.404939] env[72944]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 754.404939] env[72944]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 754.404939] env[72944]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 754.404939] env[72944]: ERROR neutron response = self.get(path=path, params=params) [ 754.405792] env[72944]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 754.405792] env[72944]: ERROR neutron raise RuntimeError(msg, last_err) [ 754.405792] env[72944]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 754.405792] env[72944]: ERROR neutron [ 757.126342] env[73072]: INFO neutron.common.config [-] Logging enabled! [ 757.126652] env[73072]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 757.126855] env[73072]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73072) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 757.127079] env[73072]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 757.127170] env[73072]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 757.127984] env[73072]: WARNING oslo_policy.policy [None req-44873d9d-ba47-464b-9f33-1a66a9a31ea9 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 757.128290] env[73072]: WARNING oslo_policy.policy [None req-44873d9d-ba47-464b-9f33-1a66a9a31ea9 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 757.129596] env[73072]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-44873d9d-ba47-464b-9f33-1a66a9a31ea9 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 757.144045] env[73072]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-44873d9d-ba47-464b-9f33-1a66a9a31ea9 None None] Session token - acquiring [ 757.144857] env[73072]: DEBUG networking_nsxv3.common.locking [None req-44873d9d-ba47-464b-9f33-1a66a9a31ea9 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73072) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 757.145068] env[73072]: DEBUG oslo_concurrency.lockutils [None req-44873d9d-ba47-464b-9f33-1a66a9a31ea9 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73072) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 757.145193] env[73072]: DEBUG oslo_concurrency.lockutils [None req-44873d9d-ba47-464b-9f33-1a66a9a31ea9 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73072) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 757.145550] env[73072]: DEBUG oslo_concurrency.lockutils [None req-44873d9d-ba47-464b-9f33-1a66a9a31ea9 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73072) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 757.154591] env[73072]: DEBUG oslo_concurrency.lockutils [None req-44873d9d-ba47-464b-9f33-1a66a9a31ea9 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73072) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 757.154951] env[73072]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-44873d9d-ba47-464b-9f33-1a66a9a31ea9 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 757.155141] env[73072]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-44873d9d-ba47-464b-9f33-1a66a9a31ea9 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73072) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 759.161219] env[73072]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-44873d9d-ba47-464b-9f33-1a66a9a31ea9 None None] Session token - acquiring [ 759.161610] env[73072]: DEBUG networking_nsxv3.common.locking [None req-44873d9d-ba47-464b-9f33-1a66a9a31ea9 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73072) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 759.161804] env[73072]: DEBUG oslo_concurrency.lockutils [None req-44873d9d-ba47-464b-9f33-1a66a9a31ea9 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73072) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 759.161930] env[73072]: DEBUG oslo_concurrency.lockutils [None req-44873d9d-ba47-464b-9f33-1a66a9a31ea9 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73072) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 759.162207] env[73072]: DEBUG oslo_concurrency.lockutils [None req-44873d9d-ba47-464b-9f33-1a66a9a31ea9 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73072) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 759.171252] env[73072]: DEBUG oslo_concurrency.lockutils [None req-44873d9d-ba47-464b-9f33-1a66a9a31ea9 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73072) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 759.171487] env[73072]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-44873d9d-ba47-464b-9f33-1a66a9a31ea9 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 759.171623] env[73072]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-44873d9d-ba47-464b-9f33-1a66a9a31ea9 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73072) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 761.172917] env[73072]: CRITICAL neutron [None req-44873d9d-ba47-464b-9f33-1a66a9a31ea9 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 761.172917] env[73072]: ERROR neutron Traceback (most recent call last): [ 761.172917] env[73072]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 761.172917] env[73072]: ERROR neutron sys.exit(main()) [ 761.172917] env[73072]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 761.172917] env[73072]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 761.172917] env[73072]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 761.172917] env[73072]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 761.172917] env[73072]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 761.172917] env[73072]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 761.172917] env[73072]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 761.172917] env[73072]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 761.172917] env[73072]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 761.172917] env[73072]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 761.172917] env[73072]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 761.172917] env[73072]: ERROR neutron response = self.get(path=path, params=params) [ 761.173823] env[73072]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 761.173823] env[73072]: ERROR neutron raise RuntimeError(msg, last_err) [ 761.173823] env[73072]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 761.173823] env[73072]: ERROR neutron [ 763.907522] env[73196]: INFO neutron.common.config [-] Logging enabled! [ 763.907835] env[73196]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 763.908212] env[73196]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73196) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 763.908447] env[73196]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 763.908491] env[73196]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 763.909353] env[73196]: WARNING oslo_policy.policy [None req-0fb197a5-6f5a-41c6-8682-9b05f68cea70 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 763.909595] env[73196]: WARNING oslo_policy.policy [None req-0fb197a5-6f5a-41c6-8682-9b05f68cea70 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 763.910985] env[73196]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-0fb197a5-6f5a-41c6-8682-9b05f68cea70 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 763.929103] env[73196]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0fb197a5-6f5a-41c6-8682-9b05f68cea70 None None] Session token - acquiring [ 763.930185] env[73196]: DEBUG networking_nsxv3.common.locking [None req-0fb197a5-6f5a-41c6-8682-9b05f68cea70 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73196) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 763.930847] env[73196]: DEBUG oslo_concurrency.lockutils [None req-0fb197a5-6f5a-41c6-8682-9b05f68cea70 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73196) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 763.931136] env[73196]: DEBUG oslo_concurrency.lockutils [None req-0fb197a5-6f5a-41c6-8682-9b05f68cea70 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73196) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 763.931650] env[73196]: DEBUG oslo_concurrency.lockutils [None req-0fb197a5-6f5a-41c6-8682-9b05f68cea70 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73196) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 763.943503] env[73196]: DEBUG oslo_concurrency.lockutils [None req-0fb197a5-6f5a-41c6-8682-9b05f68cea70 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73196) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 763.943867] env[73196]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0fb197a5-6f5a-41c6-8682-9b05f68cea70 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 763.944082] env[73196]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0fb197a5-6f5a-41c6-8682-9b05f68cea70 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73196) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 765.952392] env[73196]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0fb197a5-6f5a-41c6-8682-9b05f68cea70 None None] Session token - acquiring [ 765.952750] env[73196]: DEBUG networking_nsxv3.common.locking [None req-0fb197a5-6f5a-41c6-8682-9b05f68cea70 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73196) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 765.952976] env[73196]: DEBUG oslo_concurrency.lockutils [None req-0fb197a5-6f5a-41c6-8682-9b05f68cea70 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73196) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 765.953147] env[73196]: DEBUG oslo_concurrency.lockutils [None req-0fb197a5-6f5a-41c6-8682-9b05f68cea70 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73196) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 765.953535] env[73196]: DEBUG oslo_concurrency.lockutils [None req-0fb197a5-6f5a-41c6-8682-9b05f68cea70 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73196) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 765.963331] env[73196]: DEBUG oslo_concurrency.lockutils [None req-0fb197a5-6f5a-41c6-8682-9b05f68cea70 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73196) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 765.963586] env[73196]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0fb197a5-6f5a-41c6-8682-9b05f68cea70 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 765.963815] env[73196]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0fb197a5-6f5a-41c6-8682-9b05f68cea70 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73196) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 767.965006] env[73196]: CRITICAL neutron [None req-0fb197a5-6f5a-41c6-8682-9b05f68cea70 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 767.965006] env[73196]: ERROR neutron Traceback (most recent call last): [ 767.965006] env[73196]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 767.965006] env[73196]: ERROR neutron sys.exit(main()) [ 767.965006] env[73196]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 767.965006] env[73196]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 767.965006] env[73196]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 767.965006] env[73196]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 767.965006] env[73196]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 767.965006] env[73196]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 767.965006] env[73196]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 767.965006] env[73196]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 767.965006] env[73196]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 767.965006] env[73196]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 767.965006] env[73196]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 767.965006] env[73196]: ERROR neutron response = self.get(path=path, params=params) [ 767.965827] env[73196]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 767.965827] env[73196]: ERROR neutron raise RuntimeError(msg, last_err) [ 767.965827] env[73196]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 767.965827] env[73196]: ERROR neutron [ 770.864139] env[73329]: INFO neutron.common.config [-] Logging enabled! [ 770.864442] env[73329]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 770.864678] env[73329]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73329) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 770.864902] env[73329]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 770.865008] env[73329]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 770.865872] env[73329]: WARNING oslo_policy.policy [None req-93d9a9cb-e1d5-4956-9ac5-f2a97ede0f46 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 770.866315] env[73329]: WARNING oslo_policy.policy [None req-93d9a9cb-e1d5-4956-9ac5-f2a97ede0f46 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 770.867456] env[73329]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-93d9a9cb-e1d5-4956-9ac5-f2a97ede0f46 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 770.882661] env[73329]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-93d9a9cb-e1d5-4956-9ac5-f2a97ede0f46 None None] Session token - acquiring [ 770.883546] env[73329]: DEBUG networking_nsxv3.common.locking [None req-93d9a9cb-e1d5-4956-9ac5-f2a97ede0f46 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73329) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 770.884167] env[73329]: DEBUG oslo_concurrency.lockutils [None req-93d9a9cb-e1d5-4956-9ac5-f2a97ede0f46 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73329) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 770.884167] env[73329]: DEBUG oslo_concurrency.lockutils [None req-93d9a9cb-e1d5-4956-9ac5-f2a97ede0f46 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73329) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 770.884463] env[73329]: DEBUG oslo_concurrency.lockutils [None req-93d9a9cb-e1d5-4956-9ac5-f2a97ede0f46 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73329) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 770.893881] env[73329]: DEBUG oslo_concurrency.lockutils [None req-93d9a9cb-e1d5-4956-9ac5-f2a97ede0f46 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73329) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 770.894232] env[73329]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-93d9a9cb-e1d5-4956-9ac5-f2a97ede0f46 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 770.894436] env[73329]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-93d9a9cb-e1d5-4956-9ac5-f2a97ede0f46 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73329) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 772.900387] env[73329]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-93d9a9cb-e1d5-4956-9ac5-f2a97ede0f46 None None] Session token - acquiring [ 772.900681] env[73329]: DEBUG networking_nsxv3.common.locking [None req-93d9a9cb-e1d5-4956-9ac5-f2a97ede0f46 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73329) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 772.900857] env[73329]: DEBUG oslo_concurrency.lockutils [None req-93d9a9cb-e1d5-4956-9ac5-f2a97ede0f46 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73329) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 772.900995] env[73329]: DEBUG oslo_concurrency.lockutils [None req-93d9a9cb-e1d5-4956-9ac5-f2a97ede0f46 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73329) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 772.901300] env[73329]: DEBUG oslo_concurrency.lockutils [None req-93d9a9cb-e1d5-4956-9ac5-f2a97ede0f46 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73329) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 772.910495] env[73329]: DEBUG oslo_concurrency.lockutils [None req-93d9a9cb-e1d5-4956-9ac5-f2a97ede0f46 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73329) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 772.910736] env[73329]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-93d9a9cb-e1d5-4956-9ac5-f2a97ede0f46 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 772.910878] env[73329]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-93d9a9cb-e1d5-4956-9ac5-f2a97ede0f46 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73329) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 774.913046] env[73329]: CRITICAL neutron [None req-93d9a9cb-e1d5-4956-9ac5-f2a97ede0f46 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 774.913046] env[73329]: ERROR neutron Traceback (most recent call last): [ 774.913046] env[73329]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 774.913046] env[73329]: ERROR neutron sys.exit(main()) [ 774.913046] env[73329]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 774.913046] env[73329]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 774.913046] env[73329]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 774.913046] env[73329]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 774.913046] env[73329]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 774.913046] env[73329]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 774.913046] env[73329]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 774.913046] env[73329]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 774.913046] env[73329]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 774.913046] env[73329]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 774.913046] env[73329]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 774.913046] env[73329]: ERROR neutron response = self.get(path=path, params=params) [ 774.913880] env[73329]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 774.913880] env[73329]: ERROR neutron raise RuntimeError(msg, last_err) [ 774.913880] env[73329]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 774.913880] env[73329]: ERROR neutron [ 777.468499] env[73454]: INFO neutron.common.config [-] Logging enabled! [ 777.468747] env[73454]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 777.469006] env[73454]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73454) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 777.469292] env[73454]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 777.469331] env[73454]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 777.470159] env[73454]: WARNING oslo_policy.policy [None req-9b6d98e8-56bc-4eef-8ecb-be460001cb69 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 777.470385] env[73454]: WARNING oslo_policy.policy [None req-9b6d98e8-56bc-4eef-8ecb-be460001cb69 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 777.472559] env[73454]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-9b6d98e8-56bc-4eef-8ecb-be460001cb69 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 777.486971] env[73454]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b6d98e8-56bc-4eef-8ecb-be460001cb69 None None] Session token - acquiring [ 777.487771] env[73454]: DEBUG networking_nsxv3.common.locking [None req-9b6d98e8-56bc-4eef-8ecb-be460001cb69 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73454) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 777.487973] env[73454]: DEBUG oslo_concurrency.lockutils [None req-9b6d98e8-56bc-4eef-8ecb-be460001cb69 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73454) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 777.488123] env[73454]: DEBUG oslo_concurrency.lockutils [None req-9b6d98e8-56bc-4eef-8ecb-be460001cb69 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73454) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 777.488489] env[73454]: DEBUG oslo_concurrency.lockutils [None req-9b6d98e8-56bc-4eef-8ecb-be460001cb69 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73454) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 777.497798] env[73454]: DEBUG oslo_concurrency.lockutils [None req-9b6d98e8-56bc-4eef-8ecb-be460001cb69 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73454) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 777.498057] env[73454]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b6d98e8-56bc-4eef-8ecb-be460001cb69 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 777.498209] env[73454]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b6d98e8-56bc-4eef-8ecb-be460001cb69 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73454) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 779.504476] env[73454]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b6d98e8-56bc-4eef-8ecb-be460001cb69 None None] Session token - acquiring [ 779.504783] env[73454]: DEBUG networking_nsxv3.common.locking [None req-9b6d98e8-56bc-4eef-8ecb-be460001cb69 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73454) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 779.504980] env[73454]: DEBUG oslo_concurrency.lockutils [None req-9b6d98e8-56bc-4eef-8ecb-be460001cb69 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73454) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 779.505109] env[73454]: DEBUG oslo_concurrency.lockutils [None req-9b6d98e8-56bc-4eef-8ecb-be460001cb69 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73454) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 779.505428] env[73454]: DEBUG oslo_concurrency.lockutils [None req-9b6d98e8-56bc-4eef-8ecb-be460001cb69 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73454) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 779.515272] env[73454]: DEBUG oslo_concurrency.lockutils [None req-9b6d98e8-56bc-4eef-8ecb-be460001cb69 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73454) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 779.515561] env[73454]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b6d98e8-56bc-4eef-8ecb-be460001cb69 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 779.515708] env[73454]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b6d98e8-56bc-4eef-8ecb-be460001cb69 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73454) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 781.516883] env[73454]: CRITICAL neutron [None req-9b6d98e8-56bc-4eef-8ecb-be460001cb69 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 781.516883] env[73454]: ERROR neutron Traceback (most recent call last): [ 781.516883] env[73454]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 781.516883] env[73454]: ERROR neutron sys.exit(main()) [ 781.516883] env[73454]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 781.516883] env[73454]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 781.516883] env[73454]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 781.516883] env[73454]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 781.516883] env[73454]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 781.516883] env[73454]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 781.516883] env[73454]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 781.516883] env[73454]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 781.516883] env[73454]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 781.516883] env[73454]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 781.516883] env[73454]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 781.516883] env[73454]: ERROR neutron response = self.get(path=path, params=params) [ 781.517652] env[73454]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 781.517652] env[73454]: ERROR neutron raise RuntimeError(msg, last_err) [ 781.517652] env[73454]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 781.517652] env[73454]: ERROR neutron [ 784.321039] env[73568]: INFO neutron.common.config [-] Logging enabled! [ 784.321039] env[73568]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 784.321039] env[73568]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73568) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 784.321455] env[73568]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 784.321455] env[73568]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 784.322591] env[73568]: WARNING oslo_policy.policy [None req-f04bdd71-b425-43c0-b541-8b9a47b6ebbb None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 784.322591] env[73568]: WARNING oslo_policy.policy [None req-f04bdd71-b425-43c0-b541-8b9a47b6ebbb None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 784.323709] env[73568]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-f04bdd71-b425-43c0-b541-8b9a47b6ebbb None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 784.346223] env[73568]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f04bdd71-b425-43c0-b541-8b9a47b6ebbb None None] Session token - acquiring [ 784.346755] env[73568]: DEBUG networking_nsxv3.common.locking [None req-f04bdd71-b425-43c0-b541-8b9a47b6ebbb None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73568) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 784.347460] env[73568]: DEBUG oslo_concurrency.lockutils [None req-f04bdd71-b425-43c0-b541-8b9a47b6ebbb None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73568) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 784.347460] env[73568]: DEBUG oslo_concurrency.lockutils [None req-f04bdd71-b425-43c0-b541-8b9a47b6ebbb None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73568) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 784.347554] env[73568]: DEBUG oslo_concurrency.lockutils [None req-f04bdd71-b425-43c0-b541-8b9a47b6ebbb None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73568) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 784.358418] env[73568]: DEBUG oslo_concurrency.lockutils [None req-f04bdd71-b425-43c0-b541-8b9a47b6ebbb None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73568) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 784.358772] env[73568]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f04bdd71-b425-43c0-b541-8b9a47b6ebbb None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 784.359148] env[73568]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f04bdd71-b425-43c0-b541-8b9a47b6ebbb None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73568) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 786.365225] env[73568]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f04bdd71-b425-43c0-b541-8b9a47b6ebbb None None] Session token - acquiring [ 786.365544] env[73568]: DEBUG networking_nsxv3.common.locking [None req-f04bdd71-b425-43c0-b541-8b9a47b6ebbb None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73568) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 786.365739] env[73568]: DEBUG oslo_concurrency.lockutils [None req-f04bdd71-b425-43c0-b541-8b9a47b6ebbb None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73568) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 786.365871] env[73568]: DEBUG oslo_concurrency.lockutils [None req-f04bdd71-b425-43c0-b541-8b9a47b6ebbb None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73568) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 786.366157] env[73568]: DEBUG oslo_concurrency.lockutils [None req-f04bdd71-b425-43c0-b541-8b9a47b6ebbb None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73568) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 786.375905] env[73568]: DEBUG oslo_concurrency.lockutils [None req-f04bdd71-b425-43c0-b541-8b9a47b6ebbb None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73568) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 786.376173] env[73568]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f04bdd71-b425-43c0-b541-8b9a47b6ebbb None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 786.376434] env[73568]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f04bdd71-b425-43c0-b541-8b9a47b6ebbb None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73568) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 788.379338] env[73568]: CRITICAL neutron [None req-f04bdd71-b425-43c0-b541-8b9a47b6ebbb None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 788.379338] env[73568]: ERROR neutron Traceback (most recent call last): [ 788.379338] env[73568]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 788.379338] env[73568]: ERROR neutron sys.exit(main()) [ 788.379338] env[73568]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 788.379338] env[73568]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 788.379338] env[73568]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 788.379338] env[73568]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 788.379338] env[73568]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 788.379338] env[73568]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 788.379338] env[73568]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 788.379338] env[73568]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 788.379338] env[73568]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 788.379338] env[73568]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 788.379338] env[73568]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 788.379338] env[73568]: ERROR neutron response = self.get(path=path, params=params) [ 788.380137] env[73568]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 788.380137] env[73568]: ERROR neutron raise RuntimeError(msg, last_err) [ 788.380137] env[73568]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 788.380137] env[73568]: ERROR neutron [ 791.093516] env[73815]: INFO neutron.common.config [-] Logging enabled! [ 791.093864] env[73815]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 791.094076] env[73815]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73815) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 791.094307] env[73815]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 791.094402] env[73815]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 791.095271] env[73815]: WARNING oslo_policy.policy [None req-fa684142-bbb8-487f-b9ef-db9434de1a47 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 791.095496] env[73815]: WARNING oslo_policy.policy [None req-fa684142-bbb8-487f-b9ef-db9434de1a47 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 791.096747] env[73815]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-fa684142-bbb8-487f-b9ef-db9434de1a47 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 791.111246] env[73815]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fa684142-bbb8-487f-b9ef-db9434de1a47 None None] Session token - acquiring [ 791.112039] env[73815]: DEBUG networking_nsxv3.common.locking [None req-fa684142-bbb8-487f-b9ef-db9434de1a47 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73815) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 791.112254] env[73815]: DEBUG oslo_concurrency.lockutils [None req-fa684142-bbb8-487f-b9ef-db9434de1a47 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73815) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 791.112382] env[73815]: DEBUG oslo_concurrency.lockutils [None req-fa684142-bbb8-487f-b9ef-db9434de1a47 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73815) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 791.112922] env[73815]: DEBUG oslo_concurrency.lockutils [None req-fa684142-bbb8-487f-b9ef-db9434de1a47 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73815) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 791.123378] env[73815]: DEBUG oslo_concurrency.lockutils [None req-fa684142-bbb8-487f-b9ef-db9434de1a47 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73815) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 791.123641] env[73815]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fa684142-bbb8-487f-b9ef-db9434de1a47 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 791.123854] env[73815]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fa684142-bbb8-487f-b9ef-db9434de1a47 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73815) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 793.129870] env[73815]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fa684142-bbb8-487f-b9ef-db9434de1a47 None None] Session token - acquiring [ 793.130188] env[73815]: DEBUG networking_nsxv3.common.locking [None req-fa684142-bbb8-487f-b9ef-db9434de1a47 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73815) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 793.130379] env[73815]: DEBUG oslo_concurrency.lockutils [None req-fa684142-bbb8-487f-b9ef-db9434de1a47 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73815) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 793.130502] env[73815]: DEBUG oslo_concurrency.lockutils [None req-fa684142-bbb8-487f-b9ef-db9434de1a47 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73815) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 793.130781] env[73815]: DEBUG oslo_concurrency.lockutils [None req-fa684142-bbb8-487f-b9ef-db9434de1a47 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73815) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 793.141830] env[73815]: DEBUG oslo_concurrency.lockutils [None req-fa684142-bbb8-487f-b9ef-db9434de1a47 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73815) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 793.142281] env[73815]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fa684142-bbb8-487f-b9ef-db9434de1a47 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 793.142431] env[73815]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fa684142-bbb8-487f-b9ef-db9434de1a47 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73815) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 795.144907] env[73815]: CRITICAL neutron [None req-fa684142-bbb8-487f-b9ef-db9434de1a47 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 795.144907] env[73815]: ERROR neutron Traceback (most recent call last): [ 795.144907] env[73815]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 795.144907] env[73815]: ERROR neutron sys.exit(main()) [ 795.144907] env[73815]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 795.144907] env[73815]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 795.144907] env[73815]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 795.144907] env[73815]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 795.144907] env[73815]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 795.144907] env[73815]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 795.144907] env[73815]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 795.144907] env[73815]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 795.144907] env[73815]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 795.144907] env[73815]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 795.144907] env[73815]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 795.144907] env[73815]: ERROR neutron response = self.get(path=path, params=params) [ 795.145577] env[73815]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 795.145577] env[73815]: ERROR neutron raise RuntimeError(msg, last_err) [ 795.145577] env[73815]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 795.145577] env[73815]: ERROR neutron [ 797.887678] env[73961]: INFO neutron.common.config [-] Logging enabled! [ 797.888109] env[73961]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 797.888298] env[73961]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73961) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 797.888530] env[73961]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 797.888626] env[73961]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 797.889470] env[73961]: WARNING oslo_policy.policy [None req-d62f3b75-b9ba-44ba-8601-22966f3a9716 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 797.889743] env[73961]: WARNING oslo_policy.policy [None req-d62f3b75-b9ba-44ba-8601-22966f3a9716 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 797.891008] env[73961]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-d62f3b75-b9ba-44ba-8601-22966f3a9716 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 797.906378] env[73961]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d62f3b75-b9ba-44ba-8601-22966f3a9716 None None] Session token - acquiring [ 797.907391] env[73961]: DEBUG networking_nsxv3.common.locking [None req-d62f3b75-b9ba-44ba-8601-22966f3a9716 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73961) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 797.907596] env[73961]: DEBUG oslo_concurrency.lockutils [None req-d62f3b75-b9ba-44ba-8601-22966f3a9716 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73961) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 797.907721] env[73961]: DEBUG oslo_concurrency.lockutils [None req-d62f3b75-b9ba-44ba-8601-22966f3a9716 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73961) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 797.908109] env[73961]: DEBUG oslo_concurrency.lockutils [None req-d62f3b75-b9ba-44ba-8601-22966f3a9716 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73961) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 797.918133] env[73961]: DEBUG oslo_concurrency.lockutils [None req-d62f3b75-b9ba-44ba-8601-22966f3a9716 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73961) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 797.918403] env[73961]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d62f3b75-b9ba-44ba-8601-22966f3a9716 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 797.918557] env[73961]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d62f3b75-b9ba-44ba-8601-22966f3a9716 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73961) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 799.924504] env[73961]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d62f3b75-b9ba-44ba-8601-22966f3a9716 None None] Session token - acquiring [ 799.925145] env[73961]: DEBUG networking_nsxv3.common.locking [None req-d62f3b75-b9ba-44ba-8601-22966f3a9716 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73961) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 799.925145] env[73961]: DEBUG oslo_concurrency.lockutils [None req-d62f3b75-b9ba-44ba-8601-22966f3a9716 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73961) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 799.925259] env[73961]: DEBUG oslo_concurrency.lockutils [None req-d62f3b75-b9ba-44ba-8601-22966f3a9716 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73961) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 799.925359] env[73961]: DEBUG oslo_concurrency.lockutils [None req-d62f3b75-b9ba-44ba-8601-22966f3a9716 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73961) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 799.934397] env[73961]: DEBUG oslo_concurrency.lockutils [None req-d62f3b75-b9ba-44ba-8601-22966f3a9716 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73961) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 799.934642] env[73961]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d62f3b75-b9ba-44ba-8601-22966f3a9716 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 799.934788] env[73961]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d62f3b75-b9ba-44ba-8601-22966f3a9716 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73961) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 801.937717] env[73961]: CRITICAL neutron [None req-d62f3b75-b9ba-44ba-8601-22966f3a9716 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 801.937717] env[73961]: ERROR neutron Traceback (most recent call last): [ 801.937717] env[73961]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 801.937717] env[73961]: ERROR neutron sys.exit(main()) [ 801.937717] env[73961]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 801.937717] env[73961]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 801.937717] env[73961]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 801.937717] env[73961]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 801.937717] env[73961]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 801.937717] env[73961]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 801.937717] env[73961]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 801.937717] env[73961]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 801.937717] env[73961]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 801.937717] env[73961]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 801.937717] env[73961]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 801.937717] env[73961]: ERROR neutron response = self.get(path=path, params=params) [ 801.938602] env[73961]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 801.938602] env[73961]: ERROR neutron raise RuntimeError(msg, last_err) [ 801.938602] env[73961]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 801.938602] env[73961]: ERROR neutron [ 804.593466] env[74041]: INFO neutron.common.config [-] Logging enabled! [ 804.593884] env[74041]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 804.593982] env[74041]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74041) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 804.594197] env[74041]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 804.594290] env[74041]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 804.595093] env[74041]: WARNING oslo_policy.policy [None req-0cb9fae8-67d3-440d-a8d2-db0ebd90aae7 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 804.595314] env[74041]: WARNING oslo_policy.policy [None req-0cb9fae8-67d3-440d-a8d2-db0ebd90aae7 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 804.596508] env[74041]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-0cb9fae8-67d3-440d-a8d2-db0ebd90aae7 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 804.610252] env[74041]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0cb9fae8-67d3-440d-a8d2-db0ebd90aae7 None None] Session token - acquiring [ 804.611002] env[74041]: DEBUG networking_nsxv3.common.locking [None req-0cb9fae8-67d3-440d-a8d2-db0ebd90aae7 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74041) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 804.611190] env[74041]: DEBUG oslo_concurrency.lockutils [None req-0cb9fae8-67d3-440d-a8d2-db0ebd90aae7 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74041) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 804.611320] env[74041]: DEBUG oslo_concurrency.lockutils [None req-0cb9fae8-67d3-440d-a8d2-db0ebd90aae7 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74041) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 804.611663] env[74041]: DEBUG oslo_concurrency.lockutils [None req-0cb9fae8-67d3-440d-a8d2-db0ebd90aae7 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74041) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 804.620754] env[74041]: DEBUG oslo_concurrency.lockutils [None req-0cb9fae8-67d3-440d-a8d2-db0ebd90aae7 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74041) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 804.621035] env[74041]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0cb9fae8-67d3-440d-a8d2-db0ebd90aae7 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 804.621195] env[74041]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0cb9fae8-67d3-440d-a8d2-db0ebd90aae7 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74041) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 806.627325] env[74041]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0cb9fae8-67d3-440d-a8d2-db0ebd90aae7 None None] Session token - acquiring [ 806.627832] env[74041]: DEBUG networking_nsxv3.common.locking [None req-0cb9fae8-67d3-440d-a8d2-db0ebd90aae7 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74041) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 806.627832] env[74041]: DEBUG oslo_concurrency.lockutils [None req-0cb9fae8-67d3-440d-a8d2-db0ebd90aae7 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74041) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 806.628137] env[74041]: DEBUG oslo_concurrency.lockutils [None req-0cb9fae8-67d3-440d-a8d2-db0ebd90aae7 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74041) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 806.628334] env[74041]: DEBUG oslo_concurrency.lockutils [None req-0cb9fae8-67d3-440d-a8d2-db0ebd90aae7 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74041) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 806.637946] env[74041]: DEBUG oslo_concurrency.lockutils [None req-0cb9fae8-67d3-440d-a8d2-db0ebd90aae7 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74041) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 806.638387] env[74041]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0cb9fae8-67d3-440d-a8d2-db0ebd90aae7 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 806.638566] env[74041]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0cb9fae8-67d3-440d-a8d2-db0ebd90aae7 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74041) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 808.641110] env[74041]: CRITICAL neutron [None req-0cb9fae8-67d3-440d-a8d2-db0ebd90aae7 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 808.641110] env[74041]: ERROR neutron Traceback (most recent call last): [ 808.641110] env[74041]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 808.641110] env[74041]: ERROR neutron sys.exit(main()) [ 808.641110] env[74041]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 808.641110] env[74041]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 808.641110] env[74041]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 808.641110] env[74041]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 808.641110] env[74041]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 808.641110] env[74041]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 808.641110] env[74041]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 808.641110] env[74041]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 808.641110] env[74041]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 808.641110] env[74041]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 808.641110] env[74041]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 808.641110] env[74041]: ERROR neutron response = self.get(path=path, params=params) [ 808.641825] env[74041]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 808.641825] env[74041]: ERROR neutron raise RuntimeError(msg, last_err) [ 808.641825] env[74041]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 808.641825] env[74041]: ERROR neutron [ 811.403782] env[74220]: INFO neutron.common.config [-] Logging enabled! [ 811.404158] env[74220]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 811.404601] env[74220]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74220) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 811.404838] env[74220]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 811.404937] env[74220]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 811.405804] env[74220]: WARNING oslo_policy.policy [None req-b4ce74e0-9aaa-48bc-9039-6d31a843b19f None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 811.406053] env[74220]: WARNING oslo_policy.policy [None req-b4ce74e0-9aaa-48bc-9039-6d31a843b19f None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 811.407317] env[74220]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-b4ce74e0-9aaa-48bc-9039-6d31a843b19f None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 811.421407] env[74220]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b4ce74e0-9aaa-48bc-9039-6d31a843b19f None None] Session token - acquiring [ 811.422194] env[74220]: DEBUG networking_nsxv3.common.locking [None req-b4ce74e0-9aaa-48bc-9039-6d31a843b19f None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74220) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 811.422390] env[74220]: DEBUG oslo_concurrency.lockutils [None req-b4ce74e0-9aaa-48bc-9039-6d31a843b19f None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74220) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 811.422512] env[74220]: DEBUG oslo_concurrency.lockutils [None req-b4ce74e0-9aaa-48bc-9039-6d31a843b19f None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74220) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 811.422909] env[74220]: DEBUG oslo_concurrency.lockutils [None req-b4ce74e0-9aaa-48bc-9039-6d31a843b19f None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74220) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 811.436978] env[74220]: DEBUG oslo_concurrency.lockutils [None req-b4ce74e0-9aaa-48bc-9039-6d31a843b19f None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74220) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 811.437309] env[74220]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b4ce74e0-9aaa-48bc-9039-6d31a843b19f None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 811.437473] env[74220]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b4ce74e0-9aaa-48bc-9039-6d31a843b19f None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74220) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 813.444396] env[74220]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b4ce74e0-9aaa-48bc-9039-6d31a843b19f None None] Session token - acquiring [ 813.444396] env[74220]: DEBUG networking_nsxv3.common.locking [None req-b4ce74e0-9aaa-48bc-9039-6d31a843b19f None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74220) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 813.444396] env[74220]: DEBUG oslo_concurrency.lockutils [None req-b4ce74e0-9aaa-48bc-9039-6d31a843b19f None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74220) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 813.445103] env[74220]: DEBUG oslo_concurrency.lockutils [None req-b4ce74e0-9aaa-48bc-9039-6d31a843b19f None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74220) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 813.445103] env[74220]: DEBUG oslo_concurrency.lockutils [None req-b4ce74e0-9aaa-48bc-9039-6d31a843b19f None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74220) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 813.455177] env[74220]: DEBUG oslo_concurrency.lockutils [None req-b4ce74e0-9aaa-48bc-9039-6d31a843b19f None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74220) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 813.455440] env[74220]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b4ce74e0-9aaa-48bc-9039-6d31a843b19f None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 813.455579] env[74220]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b4ce74e0-9aaa-48bc-9039-6d31a843b19f None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74220) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 815.458541] env[74220]: CRITICAL neutron [None req-b4ce74e0-9aaa-48bc-9039-6d31a843b19f None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 815.458541] env[74220]: ERROR neutron Traceback (most recent call last): [ 815.458541] env[74220]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 815.458541] env[74220]: ERROR neutron sys.exit(main()) [ 815.458541] env[74220]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 815.458541] env[74220]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 815.458541] env[74220]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 815.458541] env[74220]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 815.458541] env[74220]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 815.458541] env[74220]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 815.458541] env[74220]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 815.458541] env[74220]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 815.458541] env[74220]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 815.458541] env[74220]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 815.458541] env[74220]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 815.458541] env[74220]: ERROR neutron response = self.get(path=path, params=params) [ 815.459210] env[74220]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 815.459210] env[74220]: ERROR neutron raise RuntimeError(msg, last_err) [ 815.459210] env[74220]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 815.459210] env[74220]: ERROR neutron [ 818.096578] env[74290]: INFO neutron.common.config [-] Logging enabled! [ 818.096919] env[74290]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 818.097185] env[74290]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74290) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 818.097358] env[74290]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 818.097454] env[74290]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 818.098892] env[74290]: WARNING oslo_policy.policy [None req-fdef71ed-5f27-4b7a-9539-297668dd7651 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 818.098892] env[74290]: WARNING oslo_policy.policy [None req-fdef71ed-5f27-4b7a-9539-297668dd7651 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 818.099831] env[74290]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-fdef71ed-5f27-4b7a-9539-297668dd7651 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 818.113578] env[74290]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fdef71ed-5f27-4b7a-9539-297668dd7651 None None] Session token - acquiring [ 818.114478] env[74290]: DEBUG networking_nsxv3.common.locking [None req-fdef71ed-5f27-4b7a-9539-297668dd7651 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74290) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 818.114688] env[74290]: DEBUG oslo_concurrency.lockutils [None req-fdef71ed-5f27-4b7a-9539-297668dd7651 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74290) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 818.114838] env[74290]: DEBUG oslo_concurrency.lockutils [None req-fdef71ed-5f27-4b7a-9539-297668dd7651 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74290) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 818.115209] env[74290]: DEBUG oslo_concurrency.lockutils [None req-fdef71ed-5f27-4b7a-9539-297668dd7651 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74290) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 818.129476] env[74290]: DEBUG oslo_concurrency.lockutils [None req-fdef71ed-5f27-4b7a-9539-297668dd7651 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74290) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 818.129763] env[74290]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fdef71ed-5f27-4b7a-9539-297668dd7651 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 818.129940] env[74290]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fdef71ed-5f27-4b7a-9539-297668dd7651 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74290) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 820.136199] env[74290]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fdef71ed-5f27-4b7a-9539-297668dd7651 None None] Session token - acquiring [ 820.136522] env[74290]: DEBUG networking_nsxv3.common.locking [None req-fdef71ed-5f27-4b7a-9539-297668dd7651 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74290) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 820.136714] env[74290]: DEBUG oslo_concurrency.lockutils [None req-fdef71ed-5f27-4b7a-9539-297668dd7651 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74290) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 820.136839] env[74290]: DEBUG oslo_concurrency.lockutils [None req-fdef71ed-5f27-4b7a-9539-297668dd7651 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74290) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 820.137130] env[74290]: DEBUG oslo_concurrency.lockutils [None req-fdef71ed-5f27-4b7a-9539-297668dd7651 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74290) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 820.146256] env[74290]: DEBUG oslo_concurrency.lockutils [None req-fdef71ed-5f27-4b7a-9539-297668dd7651 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74290) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 820.146564] env[74290]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fdef71ed-5f27-4b7a-9539-297668dd7651 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 820.146706] env[74290]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fdef71ed-5f27-4b7a-9539-297668dd7651 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74290) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 822.148926] env[74290]: CRITICAL neutron [None req-fdef71ed-5f27-4b7a-9539-297668dd7651 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 822.148926] env[74290]: ERROR neutron Traceback (most recent call last): [ 822.148926] env[74290]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 822.148926] env[74290]: ERROR neutron sys.exit(main()) [ 822.148926] env[74290]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 822.148926] env[74290]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 822.148926] env[74290]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 822.148926] env[74290]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 822.148926] env[74290]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 822.148926] env[74290]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 822.148926] env[74290]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 822.148926] env[74290]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 822.148926] env[74290]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 822.148926] env[74290]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 822.148926] env[74290]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 822.148926] env[74290]: ERROR neutron response = self.get(path=path, params=params) [ 822.149803] env[74290]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 822.149803] env[74290]: ERROR neutron raise RuntimeError(msg, last_err) [ 822.149803] env[74290]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 822.149803] env[74290]: ERROR neutron [ 824.811170] env[74394]: INFO neutron.common.config [-] Logging enabled! [ 824.811170] env[74394]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 824.811170] env[74394]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74394) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 824.811170] env[74394]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 824.811170] env[74394]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 824.811679] env[74394]: WARNING oslo_policy.policy [None req-5ae40dd8-056f-4bac-966d-372e3f4aa629 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 824.811719] env[74394]: WARNING oslo_policy.policy [None req-5ae40dd8-056f-4bac-966d-372e3f4aa629 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 824.813343] env[74394]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-5ae40dd8-056f-4bac-966d-372e3f4aa629 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 824.834703] env[74394]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5ae40dd8-056f-4bac-966d-372e3f4aa629 None None] Session token - acquiring [ 824.835605] env[74394]: DEBUG networking_nsxv3.common.locking [None req-5ae40dd8-056f-4bac-966d-372e3f4aa629 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74394) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 824.835815] env[74394]: DEBUG oslo_concurrency.lockutils [None req-5ae40dd8-056f-4bac-966d-372e3f4aa629 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74394) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 824.835939] env[74394]: DEBUG oslo_concurrency.lockutils [None req-5ae40dd8-056f-4bac-966d-372e3f4aa629 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74394) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 824.836407] env[74394]: DEBUG oslo_concurrency.lockutils [None req-5ae40dd8-056f-4bac-966d-372e3f4aa629 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74394) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 824.845794] env[74394]: DEBUG oslo_concurrency.lockutils [None req-5ae40dd8-056f-4bac-966d-372e3f4aa629 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74394) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 824.846078] env[74394]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5ae40dd8-056f-4bac-966d-372e3f4aa629 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 824.846224] env[74394]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5ae40dd8-056f-4bac-966d-372e3f4aa629 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74394) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 826.852795] env[74394]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5ae40dd8-056f-4bac-966d-372e3f4aa629 None None] Session token - acquiring [ 826.853225] env[74394]: DEBUG networking_nsxv3.common.locking [None req-5ae40dd8-056f-4bac-966d-372e3f4aa629 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74394) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 826.853501] env[74394]: DEBUG oslo_concurrency.lockutils [None req-5ae40dd8-056f-4bac-966d-372e3f4aa629 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74394) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 826.853771] env[74394]: DEBUG oslo_concurrency.lockutils [None req-5ae40dd8-056f-4bac-966d-372e3f4aa629 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74394) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 826.854114] env[74394]: DEBUG oslo_concurrency.lockutils [None req-5ae40dd8-056f-4bac-966d-372e3f4aa629 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74394) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 826.865563] env[74394]: DEBUG oslo_concurrency.lockutils [None req-5ae40dd8-056f-4bac-966d-372e3f4aa629 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74394) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 826.865959] env[74394]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5ae40dd8-056f-4bac-966d-372e3f4aa629 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 826.866112] env[74394]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5ae40dd8-056f-4bac-966d-372e3f4aa629 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74394) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 828.869113] env[74394]: CRITICAL neutron [None req-5ae40dd8-056f-4bac-966d-372e3f4aa629 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 828.869113] env[74394]: ERROR neutron Traceback (most recent call last): [ 828.869113] env[74394]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 828.869113] env[74394]: ERROR neutron sys.exit(main()) [ 828.869113] env[74394]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 828.869113] env[74394]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 828.869113] env[74394]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 828.869113] env[74394]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 828.869113] env[74394]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 828.869113] env[74394]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 828.869113] env[74394]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 828.869113] env[74394]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 828.869113] env[74394]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 828.869113] env[74394]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 828.869113] env[74394]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 828.869113] env[74394]: ERROR neutron response = self.get(path=path, params=params) [ 828.869909] env[74394]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 828.869909] env[74394]: ERROR neutron raise RuntimeError(msg, last_err) [ 828.869909] env[74394]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 828.869909] env[74394]: ERROR neutron [ 831.339895] env[74645]: INFO neutron.common.config [-] Logging enabled! [ 831.340226] env[74645]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 831.340429] env[74645]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74645) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 831.340728] env[74645]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 831.340838] env[74645]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 831.341684] env[74645]: WARNING oslo_policy.policy [None req-6bc5e351-20b7-4911-b74f-7272bcd2839d None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 831.341917] env[74645]: WARNING oslo_policy.policy [None req-6bc5e351-20b7-4911-b74f-7272bcd2839d None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 831.343095] env[74645]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-6bc5e351-20b7-4911-b74f-7272bcd2839d None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 831.357094] env[74645]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6bc5e351-20b7-4911-b74f-7272bcd2839d None None] Session token - acquiring [ 831.357843] env[74645]: DEBUG networking_nsxv3.common.locking [None req-6bc5e351-20b7-4911-b74f-7272bcd2839d None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74645) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 831.358046] env[74645]: DEBUG oslo_concurrency.lockutils [None req-6bc5e351-20b7-4911-b74f-7272bcd2839d None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74645) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 831.358168] env[74645]: DEBUG oslo_concurrency.lockutils [None req-6bc5e351-20b7-4911-b74f-7272bcd2839d None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74645) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 831.358525] env[74645]: DEBUG oslo_concurrency.lockutils [None req-6bc5e351-20b7-4911-b74f-7272bcd2839d None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74645) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 831.367375] env[74645]: DEBUG oslo_concurrency.lockutils [None req-6bc5e351-20b7-4911-b74f-7272bcd2839d None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74645) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 831.367632] env[74645]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6bc5e351-20b7-4911-b74f-7272bcd2839d None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 831.367784] env[74645]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6bc5e351-20b7-4911-b74f-7272bcd2839d None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74645) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 833.373909] env[74645]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6bc5e351-20b7-4911-b74f-7272bcd2839d None None] Session token - acquiring [ 833.374308] env[74645]: DEBUG networking_nsxv3.common.locking [None req-6bc5e351-20b7-4911-b74f-7272bcd2839d None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74645) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 833.374382] env[74645]: DEBUG oslo_concurrency.lockutils [None req-6bc5e351-20b7-4911-b74f-7272bcd2839d None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74645) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 833.374502] env[74645]: DEBUG oslo_concurrency.lockutils [None req-6bc5e351-20b7-4911-b74f-7272bcd2839d None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74645) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 833.374819] env[74645]: DEBUG oslo_concurrency.lockutils [None req-6bc5e351-20b7-4911-b74f-7272bcd2839d None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74645) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 833.383701] env[74645]: DEBUG oslo_concurrency.lockutils [None req-6bc5e351-20b7-4911-b74f-7272bcd2839d None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74645) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 833.383960] env[74645]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6bc5e351-20b7-4911-b74f-7272bcd2839d None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 833.384127] env[74645]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6bc5e351-20b7-4911-b74f-7272bcd2839d None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74645) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 835.388040] env[74645]: CRITICAL neutron [None req-6bc5e351-20b7-4911-b74f-7272bcd2839d None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 835.388040] env[74645]: ERROR neutron Traceback (most recent call last): [ 835.388040] env[74645]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 835.388040] env[74645]: ERROR neutron sys.exit(main()) [ 835.388040] env[74645]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 835.388040] env[74645]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 835.388040] env[74645]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 835.388040] env[74645]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 835.388040] env[74645]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 835.388040] env[74645]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 835.388040] env[74645]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 835.388040] env[74645]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 835.388040] env[74645]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 835.388040] env[74645]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 835.388040] env[74645]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 835.388040] env[74645]: ERROR neutron response = self.get(path=path, params=params) [ 835.388936] env[74645]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 835.388936] env[74645]: ERROR neutron raise RuntimeError(msg, last_err) [ 835.388936] env[74645]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 835.388936] env[74645]: ERROR neutron [ 837.977783] env[74658]: INFO neutron.common.config [-] Logging enabled! [ 837.978097] env[74658]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 837.978345] env[74658]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74658) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 837.978572] env[74658]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 837.978665] env[74658]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 837.979508] env[74658]: WARNING oslo_policy.policy [None req-c7ab6c5a-0890-48f9-8536-c1f7ed764091 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 837.979891] env[74658]: WARNING oslo_policy.policy [None req-c7ab6c5a-0890-48f9-8536-c1f7ed764091 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 837.981009] env[74658]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-c7ab6c5a-0890-48f9-8536-c1f7ed764091 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 837.995678] env[74658]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c7ab6c5a-0890-48f9-8536-c1f7ed764091 None None] Session token - acquiring [ 837.996535] env[74658]: DEBUG networking_nsxv3.common.locking [None req-c7ab6c5a-0890-48f9-8536-c1f7ed764091 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74658) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 837.996721] env[74658]: DEBUG oslo_concurrency.lockutils [None req-c7ab6c5a-0890-48f9-8536-c1f7ed764091 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74658) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 837.996850] env[74658]: DEBUG oslo_concurrency.lockutils [None req-c7ab6c5a-0890-48f9-8536-c1f7ed764091 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74658) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 837.997219] env[74658]: DEBUG oslo_concurrency.lockutils [None req-c7ab6c5a-0890-48f9-8536-c1f7ed764091 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74658) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 838.006449] env[74658]: DEBUG oslo_concurrency.lockutils [None req-c7ab6c5a-0890-48f9-8536-c1f7ed764091 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74658) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 838.006710] env[74658]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c7ab6c5a-0890-48f9-8536-c1f7ed764091 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 838.006864] env[74658]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c7ab6c5a-0890-48f9-8536-c1f7ed764091 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74658) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 840.012939] env[74658]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c7ab6c5a-0890-48f9-8536-c1f7ed764091 None None] Session token - acquiring [ 840.013455] env[74658]: DEBUG networking_nsxv3.common.locking [None req-c7ab6c5a-0890-48f9-8536-c1f7ed764091 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74658) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 840.013665] env[74658]: DEBUG oslo_concurrency.lockutils [None req-c7ab6c5a-0890-48f9-8536-c1f7ed764091 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74658) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 840.013823] env[74658]: DEBUG oslo_concurrency.lockutils [None req-c7ab6c5a-0890-48f9-8536-c1f7ed764091 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74658) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 840.014099] env[74658]: DEBUG oslo_concurrency.lockutils [None req-c7ab6c5a-0890-48f9-8536-c1f7ed764091 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74658) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 840.023532] env[74658]: DEBUG oslo_concurrency.lockutils [None req-c7ab6c5a-0890-48f9-8536-c1f7ed764091 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74658) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 840.023788] env[74658]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c7ab6c5a-0890-48f9-8536-c1f7ed764091 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 840.023908] env[74658]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c7ab6c5a-0890-48f9-8536-c1f7ed764091 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74658) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 842.024922] env[74658]: CRITICAL neutron [None req-c7ab6c5a-0890-48f9-8536-c1f7ed764091 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 842.024922] env[74658]: ERROR neutron Traceback (most recent call last): [ 842.024922] env[74658]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 842.024922] env[74658]: ERROR neutron sys.exit(main()) [ 842.024922] env[74658]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 842.024922] env[74658]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 842.024922] env[74658]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 842.024922] env[74658]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 842.024922] env[74658]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 842.024922] env[74658]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 842.024922] env[74658]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 842.024922] env[74658]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 842.024922] env[74658]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 842.024922] env[74658]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 842.024922] env[74658]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 842.024922] env[74658]: ERROR neutron response = self.get(path=path, params=params) [ 842.025639] env[74658]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 842.025639] env[74658]: ERROR neutron raise RuntimeError(msg, last_err) [ 842.025639] env[74658]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 842.025639] env[74658]: ERROR neutron [ 844.484130] env[74709]: INFO neutron.common.config [-] Logging enabled! [ 844.484465] env[74709]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 844.484669] env[74709]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74709) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 844.484928] env[74709]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 844.485019] env[74709]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 844.485937] env[74709]: WARNING oslo_policy.policy [None req-63e4eca2-5326-42cc-b7f6-6b7268d04600 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 844.486083] env[74709]: WARNING oslo_policy.policy [None req-63e4eca2-5326-42cc-b7f6-6b7268d04600 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 844.487346] env[74709]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-63e4eca2-5326-42cc-b7f6-6b7268d04600 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 844.508795] env[74709]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63e4eca2-5326-42cc-b7f6-6b7268d04600 None None] Session token - acquiring [ 844.509668] env[74709]: DEBUG networking_nsxv3.common.locking [None req-63e4eca2-5326-42cc-b7f6-6b7268d04600 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74709) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 844.509876] env[74709]: DEBUG oslo_concurrency.lockutils [None req-63e4eca2-5326-42cc-b7f6-6b7268d04600 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74709) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 844.510006] env[74709]: DEBUG oslo_concurrency.lockutils [None req-63e4eca2-5326-42cc-b7f6-6b7268d04600 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74709) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 844.510384] env[74709]: DEBUG oslo_concurrency.lockutils [None req-63e4eca2-5326-42cc-b7f6-6b7268d04600 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74709) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 844.519480] env[74709]: DEBUG oslo_concurrency.lockutils [None req-63e4eca2-5326-42cc-b7f6-6b7268d04600 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74709) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 844.519730] env[74709]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63e4eca2-5326-42cc-b7f6-6b7268d04600 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 844.519881] env[74709]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63e4eca2-5326-42cc-b7f6-6b7268d04600 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74709) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 846.526018] env[74709]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63e4eca2-5326-42cc-b7f6-6b7268d04600 None None] Session token - acquiring [ 846.526338] env[74709]: DEBUG networking_nsxv3.common.locking [None req-63e4eca2-5326-42cc-b7f6-6b7268d04600 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74709) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 846.526476] env[74709]: DEBUG oslo_concurrency.lockutils [None req-63e4eca2-5326-42cc-b7f6-6b7268d04600 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74709) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 846.526597] env[74709]: DEBUG oslo_concurrency.lockutils [None req-63e4eca2-5326-42cc-b7f6-6b7268d04600 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74709) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 846.526872] env[74709]: DEBUG oslo_concurrency.lockutils [None req-63e4eca2-5326-42cc-b7f6-6b7268d04600 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74709) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 846.536170] env[74709]: DEBUG oslo_concurrency.lockutils [None req-63e4eca2-5326-42cc-b7f6-6b7268d04600 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74709) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 846.536418] env[74709]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63e4eca2-5326-42cc-b7f6-6b7268d04600 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 846.536555] env[74709]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63e4eca2-5326-42cc-b7f6-6b7268d04600 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74709) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 848.539506] env[74709]: CRITICAL neutron [None req-63e4eca2-5326-42cc-b7f6-6b7268d04600 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 848.539506] env[74709]: ERROR neutron Traceback (most recent call last): [ 848.539506] env[74709]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 848.539506] env[74709]: ERROR neutron sys.exit(main()) [ 848.539506] env[74709]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 848.539506] env[74709]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 848.539506] env[74709]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 848.539506] env[74709]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 848.539506] env[74709]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 848.539506] env[74709]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 848.539506] env[74709]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 848.539506] env[74709]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 848.539506] env[74709]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 848.539506] env[74709]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 848.539506] env[74709]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 848.539506] env[74709]: ERROR neutron response = self.get(path=path, params=params) [ 848.540414] env[74709]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 848.540414] env[74709]: ERROR neutron raise RuntimeError(msg, last_err) [ 848.540414] env[74709]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 848.540414] env[74709]: ERROR neutron [ 851.017977] env[74801]: INFO neutron.common.config [-] Logging enabled! [ 851.018281] env[74801]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 851.018519] env[74801]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74801) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 851.018731] env[74801]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 851.018829] env[74801]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 851.019670] env[74801]: WARNING oslo_policy.policy [None req-b66d2859-d688-4526-aa3b-76ec6978a181 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 851.019892] env[74801]: WARNING oslo_policy.policy [None req-b66d2859-d688-4526-aa3b-76ec6978a181 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 851.021263] env[74801]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-b66d2859-d688-4526-aa3b-76ec6978a181 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 851.035919] env[74801]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b66d2859-d688-4526-aa3b-76ec6978a181 None None] Session token - acquiring [ 851.036718] env[74801]: DEBUG networking_nsxv3.common.locking [None req-b66d2859-d688-4526-aa3b-76ec6978a181 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74801) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 851.036921] env[74801]: DEBUG oslo_concurrency.lockutils [None req-b66d2859-d688-4526-aa3b-76ec6978a181 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74801) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 851.037048] env[74801]: DEBUG oslo_concurrency.lockutils [None req-b66d2859-d688-4526-aa3b-76ec6978a181 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74801) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 851.037416] env[74801]: DEBUG oslo_concurrency.lockutils [None req-b66d2859-d688-4526-aa3b-76ec6978a181 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74801) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 851.046128] env[74801]: DEBUG oslo_concurrency.lockutils [None req-b66d2859-d688-4526-aa3b-76ec6978a181 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74801) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 851.046367] env[74801]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b66d2859-d688-4526-aa3b-76ec6978a181 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 851.046519] env[74801]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b66d2859-d688-4526-aa3b-76ec6978a181 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74801) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 853.052601] env[74801]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b66d2859-d688-4526-aa3b-76ec6978a181 None None] Session token - acquiring [ 853.052981] env[74801]: DEBUG networking_nsxv3.common.locking [None req-b66d2859-d688-4526-aa3b-76ec6978a181 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74801) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 853.053184] env[74801]: DEBUG oslo_concurrency.lockutils [None req-b66d2859-d688-4526-aa3b-76ec6978a181 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74801) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 853.053322] env[74801]: DEBUG oslo_concurrency.lockutils [None req-b66d2859-d688-4526-aa3b-76ec6978a181 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74801) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 853.053611] env[74801]: DEBUG oslo_concurrency.lockutils [None req-b66d2859-d688-4526-aa3b-76ec6978a181 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74801) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 853.062633] env[74801]: DEBUG oslo_concurrency.lockutils [None req-b66d2859-d688-4526-aa3b-76ec6978a181 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74801) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 853.062863] env[74801]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b66d2859-d688-4526-aa3b-76ec6978a181 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 853.063005] env[74801]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b66d2859-d688-4526-aa3b-76ec6978a181 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74801) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 855.066005] env[74801]: CRITICAL neutron [None req-b66d2859-d688-4526-aa3b-76ec6978a181 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 855.066005] env[74801]: ERROR neutron Traceback (most recent call last): [ 855.066005] env[74801]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 855.066005] env[74801]: ERROR neutron sys.exit(main()) [ 855.066005] env[74801]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 855.066005] env[74801]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 855.066005] env[74801]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 855.066005] env[74801]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 855.066005] env[74801]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 855.066005] env[74801]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 855.066005] env[74801]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 855.066005] env[74801]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 855.066005] env[74801]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 855.066005] env[74801]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 855.066005] env[74801]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 855.066005] env[74801]: ERROR neutron response = self.get(path=path, params=params) [ 855.066860] env[74801]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 855.066860] env[74801]: ERROR neutron raise RuntimeError(msg, last_err) [ 855.066860] env[74801]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 855.066860] env[74801]: ERROR neutron [ 858.323380] env[74892]: INFO neutron.common.config [-] Logging enabled! [ 858.323681] env[74892]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 858.323960] env[74892]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74892) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 858.324210] env[74892]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 858.324308] env[74892]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 858.325397] env[74892]: WARNING oslo_policy.policy [None req-426fe779-5730-48c2-b84d-f30ebedfd6b7 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 858.325620] env[74892]: WARNING oslo_policy.policy [None req-426fe779-5730-48c2-b84d-f30ebedfd6b7 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 858.326843] env[74892]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-426fe779-5730-48c2-b84d-f30ebedfd6b7 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 858.340734] env[74892]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-426fe779-5730-48c2-b84d-f30ebedfd6b7 None None] Session token - acquiring [ 858.341510] env[74892]: DEBUG networking_nsxv3.common.locking [None req-426fe779-5730-48c2-b84d-f30ebedfd6b7 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74892) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 858.341716] env[74892]: DEBUG oslo_concurrency.lockutils [None req-426fe779-5730-48c2-b84d-f30ebedfd6b7 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74892) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 858.341865] env[74892]: DEBUG oslo_concurrency.lockutils [None req-426fe779-5730-48c2-b84d-f30ebedfd6b7 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74892) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 858.342292] env[74892]: DEBUG oslo_concurrency.lockutils [None req-426fe779-5730-48c2-b84d-f30ebedfd6b7 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74892) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 858.351166] env[74892]: DEBUG oslo_concurrency.lockutils [None req-426fe779-5730-48c2-b84d-f30ebedfd6b7 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74892) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 858.351416] env[74892]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-426fe779-5730-48c2-b84d-f30ebedfd6b7 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 858.351566] env[74892]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-426fe779-5730-48c2-b84d-f30ebedfd6b7 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74892) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 860.357889] env[74892]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-426fe779-5730-48c2-b84d-f30ebedfd6b7 None None] Session token - acquiring [ 860.358265] env[74892]: DEBUG networking_nsxv3.common.locking [None req-426fe779-5730-48c2-b84d-f30ebedfd6b7 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74892) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 860.358407] env[74892]: DEBUG oslo_concurrency.lockutils [None req-426fe779-5730-48c2-b84d-f30ebedfd6b7 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74892) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 860.358545] env[74892]: DEBUG oslo_concurrency.lockutils [None req-426fe779-5730-48c2-b84d-f30ebedfd6b7 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74892) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 860.358827] env[74892]: DEBUG oslo_concurrency.lockutils [None req-426fe779-5730-48c2-b84d-f30ebedfd6b7 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74892) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 860.367681] env[74892]: DEBUG oslo_concurrency.lockutils [None req-426fe779-5730-48c2-b84d-f30ebedfd6b7 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74892) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 860.367915] env[74892]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-426fe779-5730-48c2-b84d-f30ebedfd6b7 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 860.368073] env[74892]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-426fe779-5730-48c2-b84d-f30ebedfd6b7 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74892) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 862.369646] env[74892]: CRITICAL neutron [None req-426fe779-5730-48c2-b84d-f30ebedfd6b7 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 862.369646] env[74892]: ERROR neutron Traceback (most recent call last): [ 862.369646] env[74892]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 862.369646] env[74892]: ERROR neutron sys.exit(main()) [ 862.369646] env[74892]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 862.369646] env[74892]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 862.369646] env[74892]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 862.369646] env[74892]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 862.369646] env[74892]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 862.369646] env[74892]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 862.369646] env[74892]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 862.369646] env[74892]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 862.369646] env[74892]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 862.369646] env[74892]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 862.369646] env[74892]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 862.369646] env[74892]: ERROR neutron response = self.get(path=path, params=params) [ 862.370282] env[74892]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 862.370282] env[74892]: ERROR neutron raise RuntimeError(msg, last_err) [ 862.370282] env[74892]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 862.370282] env[74892]: ERROR neutron [ 865.002127] env[74924]: INFO neutron.common.config [-] Logging enabled! [ 865.002726] env[74924]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 865.002726] env[74924]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74924) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 865.002913] env[74924]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 865.003012] env[74924]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 865.003876] env[74924]: WARNING oslo_policy.policy [None req-ed4434e9-773e-467d-a3b0-91c7bcedbe15 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 865.004626] env[74924]: WARNING oslo_policy.policy [None req-ed4434e9-773e-467d-a3b0-91c7bcedbe15 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 865.005382] env[74924]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-ed4434e9-773e-467d-a3b0-91c7bcedbe15 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 865.021204] env[74924]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ed4434e9-773e-467d-a3b0-91c7bcedbe15 None None] Session token - acquiring [ 865.021204] env[74924]: DEBUG networking_nsxv3.common.locking [None req-ed4434e9-773e-467d-a3b0-91c7bcedbe15 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74924) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 865.021204] env[74924]: DEBUG oslo_concurrency.lockutils [None req-ed4434e9-773e-467d-a3b0-91c7bcedbe15 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74924) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 865.021342] env[74924]: DEBUG oslo_concurrency.lockutils [None req-ed4434e9-773e-467d-a3b0-91c7bcedbe15 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74924) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 865.021342] env[74924]: DEBUG oslo_concurrency.lockutils [None req-ed4434e9-773e-467d-a3b0-91c7bcedbe15 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74924) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 865.030451] env[74924]: DEBUG oslo_concurrency.lockutils [None req-ed4434e9-773e-467d-a3b0-91c7bcedbe15 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74924) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 865.030711] env[74924]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ed4434e9-773e-467d-a3b0-91c7bcedbe15 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 865.030864] env[74924]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ed4434e9-773e-467d-a3b0-91c7bcedbe15 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74924) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 867.039130] env[74924]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ed4434e9-773e-467d-a3b0-91c7bcedbe15 None None] Session token - acquiring [ 867.039130] env[74924]: DEBUG networking_nsxv3.common.locking [None req-ed4434e9-773e-467d-a3b0-91c7bcedbe15 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74924) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 867.039130] env[74924]: DEBUG oslo_concurrency.lockutils [None req-ed4434e9-773e-467d-a3b0-91c7bcedbe15 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74924) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 867.039469] env[74924]: DEBUG oslo_concurrency.lockutils [None req-ed4434e9-773e-467d-a3b0-91c7bcedbe15 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74924) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 867.039469] env[74924]: DEBUG oslo_concurrency.lockutils [None req-ed4434e9-773e-467d-a3b0-91c7bcedbe15 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74924) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 867.047594] env[74924]: DEBUG oslo_concurrency.lockutils [None req-ed4434e9-773e-467d-a3b0-91c7bcedbe15 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74924) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 867.047594] env[74924]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ed4434e9-773e-467d-a3b0-91c7bcedbe15 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 867.047752] env[74924]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ed4434e9-773e-467d-a3b0-91c7bcedbe15 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74924) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 869.051326] env[74924]: CRITICAL neutron [None req-ed4434e9-773e-467d-a3b0-91c7bcedbe15 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 869.051326] env[74924]: ERROR neutron Traceback (most recent call last): [ 869.051326] env[74924]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 869.051326] env[74924]: ERROR neutron sys.exit(main()) [ 869.051326] env[74924]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 869.051326] env[74924]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 869.051326] env[74924]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 869.051326] env[74924]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 869.051326] env[74924]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 869.051326] env[74924]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 869.051326] env[74924]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 869.051326] env[74924]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 869.051326] env[74924]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 869.051326] env[74924]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 869.051326] env[74924]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 869.051326] env[74924]: ERROR neutron response = self.get(path=path, params=params) [ 869.051979] env[74924]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 869.051979] env[74924]: ERROR neutron raise RuntimeError(msg, last_err) [ 869.051979] env[74924]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 869.051979] env[74924]: ERROR neutron [ 871.658280] env[74987]: INFO neutron.common.config [-] Logging enabled! [ 871.658593] env[74987]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 871.658851] env[74987]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74987) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 871.659127] env[74987]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 871.659226] env[74987]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 871.660159] env[74987]: WARNING oslo_policy.policy [None req-fa96266d-fe2f-430c-8983-05ed8b76aeba None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 871.660392] env[74987]: WARNING oslo_policy.policy [None req-fa96266d-fe2f-430c-8983-05ed8b76aeba None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 871.662030] env[74987]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-fa96266d-fe2f-430c-8983-05ed8b76aeba None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 871.681498] env[74987]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fa96266d-fe2f-430c-8983-05ed8b76aeba None None] Session token - acquiring [ 871.682377] env[74987]: DEBUG networking_nsxv3.common.locking [None req-fa96266d-fe2f-430c-8983-05ed8b76aeba None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74987) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 871.682586] env[74987]: DEBUG oslo_concurrency.lockutils [None req-fa96266d-fe2f-430c-8983-05ed8b76aeba None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74987) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 871.682714] env[74987]: DEBUG oslo_concurrency.lockutils [None req-fa96266d-fe2f-430c-8983-05ed8b76aeba None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74987) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 871.683089] env[74987]: DEBUG oslo_concurrency.lockutils [None req-fa96266d-fe2f-430c-8983-05ed8b76aeba None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74987) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 871.692914] env[74987]: DEBUG oslo_concurrency.lockutils [None req-fa96266d-fe2f-430c-8983-05ed8b76aeba None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74987) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 871.693190] env[74987]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fa96266d-fe2f-430c-8983-05ed8b76aeba None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 871.693358] env[74987]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fa96266d-fe2f-430c-8983-05ed8b76aeba None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74987) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 873.699376] env[74987]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fa96266d-fe2f-430c-8983-05ed8b76aeba None None] Session token - acquiring [ 873.699742] env[74987]: DEBUG networking_nsxv3.common.locking [None req-fa96266d-fe2f-430c-8983-05ed8b76aeba None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74987) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 873.699899] env[74987]: DEBUG oslo_concurrency.lockutils [None req-fa96266d-fe2f-430c-8983-05ed8b76aeba None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74987) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 873.700069] env[74987]: DEBUG oslo_concurrency.lockutils [None req-fa96266d-fe2f-430c-8983-05ed8b76aeba None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74987) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 873.700340] env[74987]: DEBUG oslo_concurrency.lockutils [None req-fa96266d-fe2f-430c-8983-05ed8b76aeba None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74987) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 873.710046] env[74987]: DEBUG oslo_concurrency.lockutils [None req-fa96266d-fe2f-430c-8983-05ed8b76aeba None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74987) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 873.710285] env[74987]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fa96266d-fe2f-430c-8983-05ed8b76aeba None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 873.710421] env[74987]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fa96266d-fe2f-430c-8983-05ed8b76aeba None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74987) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 875.712975] env[74987]: CRITICAL neutron [None req-fa96266d-fe2f-430c-8983-05ed8b76aeba None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 875.712975] env[74987]: ERROR neutron Traceback (most recent call last): [ 875.712975] env[74987]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 875.712975] env[74987]: ERROR neutron sys.exit(main()) [ 875.712975] env[74987]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 875.712975] env[74987]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 875.712975] env[74987]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 875.712975] env[74987]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 875.712975] env[74987]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 875.712975] env[74987]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 875.712975] env[74987]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 875.712975] env[74987]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 875.712975] env[74987]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 875.712975] env[74987]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 875.712975] env[74987]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 875.712975] env[74987]: ERROR neutron response = self.get(path=path, params=params) [ 875.713615] env[74987]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 875.713615] env[74987]: ERROR neutron raise RuntimeError(msg, last_err) [ 875.713615] env[74987]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 875.713615] env[74987]: ERROR neutron [ 878.457960] env[75242]: INFO neutron.common.config [-] Logging enabled! [ 878.458263] env[75242]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 878.458367] env[75242]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=75242) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 878.458596] env[75242]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 878.458692] env[75242]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 878.460913] env[75242]: WARNING oslo_policy.policy [None req-ad8ab082-fe1d-410a-b758-c466283e750a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 878.460913] env[75242]: WARNING oslo_policy.policy [None req-ad8ab082-fe1d-410a-b758-c466283e750a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 878.461292] env[75242]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-ad8ab082-fe1d-410a-b758-c466283e750a None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 878.479276] env[75242]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ad8ab082-fe1d-410a-b758-c466283e750a None None] Session token - acquiring [ 878.480069] env[75242]: DEBUG networking_nsxv3.common.locking [None req-ad8ab082-fe1d-410a-b758-c466283e750a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75242) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 878.480310] env[75242]: DEBUG oslo_concurrency.lockutils [None req-ad8ab082-fe1d-410a-b758-c466283e750a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75242) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 878.480446] env[75242]: DEBUG oslo_concurrency.lockutils [None req-ad8ab082-fe1d-410a-b758-c466283e750a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75242) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 878.480822] env[75242]: DEBUG oslo_concurrency.lockutils [None req-ad8ab082-fe1d-410a-b758-c466283e750a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75242) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 878.492067] env[75242]: DEBUG oslo_concurrency.lockutils [None req-ad8ab082-fe1d-410a-b758-c466283e750a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75242) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 878.492381] env[75242]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ad8ab082-fe1d-410a-b758-c466283e750a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 878.493047] env[75242]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ad8ab082-fe1d-410a-b758-c466283e750a None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=75242) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 880.500482] env[75242]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ad8ab082-fe1d-410a-b758-c466283e750a None None] Session token - acquiring [ 880.500843] env[75242]: DEBUG networking_nsxv3.common.locking [None req-ad8ab082-fe1d-410a-b758-c466283e750a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75242) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 880.501050] env[75242]: DEBUG oslo_concurrency.lockutils [None req-ad8ab082-fe1d-410a-b758-c466283e750a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75242) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 880.501175] env[75242]: DEBUG oslo_concurrency.lockutils [None req-ad8ab082-fe1d-410a-b758-c466283e750a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75242) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 880.501468] env[75242]: DEBUG oslo_concurrency.lockutils [None req-ad8ab082-fe1d-410a-b758-c466283e750a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75242) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 880.511049] env[75242]: DEBUG oslo_concurrency.lockutils [None req-ad8ab082-fe1d-410a-b758-c466283e750a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75242) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 880.511302] env[75242]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ad8ab082-fe1d-410a-b758-c466283e750a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 880.511472] env[75242]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ad8ab082-fe1d-410a-b758-c466283e750a None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=75242) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 882.514109] env[75242]: CRITICAL neutron [None req-ad8ab082-fe1d-410a-b758-c466283e750a None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 882.514109] env[75242]: ERROR neutron Traceback (most recent call last): [ 882.514109] env[75242]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 882.514109] env[75242]: ERROR neutron sys.exit(main()) [ 882.514109] env[75242]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 882.514109] env[75242]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 882.514109] env[75242]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 882.514109] env[75242]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 882.514109] env[75242]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 882.514109] env[75242]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 882.514109] env[75242]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 882.514109] env[75242]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 882.514109] env[75242]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 882.514109] env[75242]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 882.514109] env[75242]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 882.514109] env[75242]: ERROR neutron response = self.get(path=path, params=params) [ 882.514844] env[75242]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 882.514844] env[75242]: ERROR neutron raise RuntimeError(msg, last_err) [ 882.514844] env[75242]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 882.514844] env[75242]: ERROR neutron [ 885.319076] env[75374]: INFO neutron.common.config [-] Logging enabled! [ 885.319393] env[75374]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 885.319625] env[75374]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=75374) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 885.319904] env[75374]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 885.319971] env[75374]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 885.320822] env[75374]: WARNING oslo_policy.policy [None req-5d483a2d-467c-433d-896d-622c498ed07d None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 885.321049] env[75374]: WARNING oslo_policy.policy [None req-5d483a2d-467c-433d-896d-622c498ed07d None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 885.322292] env[75374]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-5d483a2d-467c-433d-896d-622c498ed07d None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 885.336518] env[75374]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5d483a2d-467c-433d-896d-622c498ed07d None None] Session token - acquiring [ 885.337294] env[75374]: DEBUG networking_nsxv3.common.locking [None req-5d483a2d-467c-433d-896d-622c498ed07d None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75374) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 885.337498] env[75374]: DEBUG oslo_concurrency.lockutils [None req-5d483a2d-467c-433d-896d-622c498ed07d None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75374) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 885.337627] env[75374]: DEBUG oslo_concurrency.lockutils [None req-5d483a2d-467c-433d-896d-622c498ed07d None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75374) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 885.338064] env[75374]: DEBUG oslo_concurrency.lockutils [None req-5d483a2d-467c-433d-896d-622c498ed07d None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75374) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 885.346896] env[75374]: DEBUG oslo_concurrency.lockutils [None req-5d483a2d-467c-433d-896d-622c498ed07d None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75374) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 885.347155] env[75374]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5d483a2d-467c-433d-896d-622c498ed07d None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 885.347304] env[75374]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5d483a2d-467c-433d-896d-622c498ed07d None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=75374) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 887.353449] env[75374]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5d483a2d-467c-433d-896d-622c498ed07d None None] Session token - acquiring [ 887.353884] env[75374]: DEBUG networking_nsxv3.common.locking [None req-5d483a2d-467c-433d-896d-622c498ed07d None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75374) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 887.354103] env[75374]: DEBUG oslo_concurrency.lockutils [None req-5d483a2d-467c-433d-896d-622c498ed07d None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75374) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 887.354280] env[75374]: DEBUG oslo_concurrency.lockutils [None req-5d483a2d-467c-433d-896d-622c498ed07d None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75374) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 887.354510] env[75374]: DEBUG oslo_concurrency.lockutils [None req-5d483a2d-467c-433d-896d-622c498ed07d None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75374) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 887.367916] env[75374]: DEBUG oslo_concurrency.lockutils [None req-5d483a2d-467c-433d-896d-622c498ed07d None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75374) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 887.368879] env[75374]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5d483a2d-467c-433d-896d-622c498ed07d None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 887.369037] env[75374]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5d483a2d-467c-433d-896d-622c498ed07d None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=75374) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 889.371950] env[75374]: CRITICAL neutron [None req-5d483a2d-467c-433d-896d-622c498ed07d None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 889.371950] env[75374]: ERROR neutron Traceback (most recent call last): [ 889.371950] env[75374]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 889.371950] env[75374]: ERROR neutron sys.exit(main()) [ 889.371950] env[75374]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 889.371950] env[75374]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 889.371950] env[75374]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 889.371950] env[75374]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 889.371950] env[75374]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 889.371950] env[75374]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 889.371950] env[75374]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 889.371950] env[75374]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 889.371950] env[75374]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 889.371950] env[75374]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 889.371950] env[75374]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 889.371950] env[75374]: ERROR neutron response = self.get(path=path, params=params) [ 889.372666] env[75374]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 889.372666] env[75374]: ERROR neutron raise RuntimeError(msg, last_err) [ 889.372666] env[75374]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 889.372666] env[75374]: ERROR neutron [ 892.058282] env[75610]: INFO neutron.common.config [-] Logging enabled! [ 892.058618] env[75610]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 892.058833] env[75610]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=75610) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 892.059063] env[75610]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 892.059156] env[75610]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 892.059998] env[75610]: WARNING oslo_policy.policy [None req-0b324d2b-f88f-4ee9-a364-611e59ad07d6 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 892.060241] env[75610]: WARNING oslo_policy.policy [None req-0b324d2b-f88f-4ee9-a364-611e59ad07d6 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 892.061618] env[75610]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-0b324d2b-f88f-4ee9-a364-611e59ad07d6 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 892.076067] env[75610]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0b324d2b-f88f-4ee9-a364-611e59ad07d6 None None] Session token - acquiring [ 892.076956] env[75610]: DEBUG networking_nsxv3.common.locking [None req-0b324d2b-f88f-4ee9-a364-611e59ad07d6 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75610) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 892.077246] env[75610]: DEBUG oslo_concurrency.lockutils [None req-0b324d2b-f88f-4ee9-a364-611e59ad07d6 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75610) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 892.077427] env[75610]: DEBUG oslo_concurrency.lockutils [None req-0b324d2b-f88f-4ee9-a364-611e59ad07d6 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75610) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 892.077853] env[75610]: DEBUG oslo_concurrency.lockutils [None req-0b324d2b-f88f-4ee9-a364-611e59ad07d6 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75610) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 892.087236] env[75610]: DEBUG oslo_concurrency.lockutils [None req-0b324d2b-f88f-4ee9-a364-611e59ad07d6 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75610) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 892.087565] env[75610]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0b324d2b-f88f-4ee9-a364-611e59ad07d6 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 892.087775] env[75610]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0b324d2b-f88f-4ee9-a364-611e59ad07d6 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=75610) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 894.094316] env[75610]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0b324d2b-f88f-4ee9-a364-611e59ad07d6 None None] Session token - acquiring [ 894.094566] env[75610]: DEBUG networking_nsxv3.common.locking [None req-0b324d2b-f88f-4ee9-a364-611e59ad07d6 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75610) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 894.094566] env[75610]: DEBUG oslo_concurrency.lockutils [None req-0b324d2b-f88f-4ee9-a364-611e59ad07d6 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75610) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 894.094714] env[75610]: DEBUG oslo_concurrency.lockutils [None req-0b324d2b-f88f-4ee9-a364-611e59ad07d6 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75610) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 894.096268] env[75610]: DEBUG oslo_concurrency.lockutils [None req-0b324d2b-f88f-4ee9-a364-611e59ad07d6 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75610) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 894.105295] env[75610]: DEBUG oslo_concurrency.lockutils [None req-0b324d2b-f88f-4ee9-a364-611e59ad07d6 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75610) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 894.105295] env[75610]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0b324d2b-f88f-4ee9-a364-611e59ad07d6 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 894.105899] env[75610]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0b324d2b-f88f-4ee9-a364-611e59ad07d6 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=75610) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 896.108322] env[75610]: CRITICAL neutron [None req-0b324d2b-f88f-4ee9-a364-611e59ad07d6 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 896.108322] env[75610]: ERROR neutron Traceback (most recent call last): [ 896.108322] env[75610]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 896.108322] env[75610]: ERROR neutron sys.exit(main()) [ 896.108322] env[75610]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 896.108322] env[75610]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 896.108322] env[75610]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 896.108322] env[75610]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 896.108322] env[75610]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 896.108322] env[75610]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 896.108322] env[75610]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 896.108322] env[75610]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 896.108322] env[75610]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 896.108322] env[75610]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 896.108322] env[75610]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 896.108322] env[75610]: ERROR neutron response = self.get(path=path, params=params) [ 896.109172] env[75610]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 896.109172] env[75610]: ERROR neutron raise RuntimeError(msg, last_err) [ 896.109172] env[75610]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 896.109172] env[75610]: ERROR neutron [ 898.947250] env[75725]: INFO neutron.common.config [-] Logging enabled! [ 898.947523] env[75725]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 898.947792] env[75725]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=75725) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 898.948063] env[75725]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 898.948173] env[75725]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 898.949203] env[75725]: WARNING oslo_policy.policy [None req-a681c6b4-3785-4df1-b39e-ba0e96fde057 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 898.949288] env[75725]: WARNING oslo_policy.policy [None req-a681c6b4-3785-4df1-b39e-ba0e96fde057 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 898.950541] env[75725]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-a681c6b4-3785-4df1-b39e-ba0e96fde057 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 898.966034] env[75725]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a681c6b4-3785-4df1-b39e-ba0e96fde057 None None] Session token - acquiring [ 898.966274] env[75725]: DEBUG networking_nsxv3.common.locking [None req-a681c6b4-3785-4df1-b39e-ba0e96fde057 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75725) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 898.966505] env[75725]: DEBUG oslo_concurrency.lockutils [None req-a681c6b4-3785-4df1-b39e-ba0e96fde057 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75725) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 898.966644] env[75725]: DEBUG oslo_concurrency.lockutils [None req-a681c6b4-3785-4df1-b39e-ba0e96fde057 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75725) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 898.967073] env[75725]: DEBUG oslo_concurrency.lockutils [None req-a681c6b4-3785-4df1-b39e-ba0e96fde057 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75725) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 898.976666] env[75725]: DEBUG oslo_concurrency.lockutils [None req-a681c6b4-3785-4df1-b39e-ba0e96fde057 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75725) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 898.976955] env[75725]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a681c6b4-3785-4df1-b39e-ba0e96fde057 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 898.977125] env[75725]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a681c6b4-3785-4df1-b39e-ba0e96fde057 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=75725) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 900.983035] env[75725]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a681c6b4-3785-4df1-b39e-ba0e96fde057 None None] Session token - acquiring [ 900.983436] env[75725]: DEBUG networking_nsxv3.common.locking [None req-a681c6b4-3785-4df1-b39e-ba0e96fde057 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75725) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 900.983585] env[75725]: DEBUG oslo_concurrency.lockutils [None req-a681c6b4-3785-4df1-b39e-ba0e96fde057 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75725) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 900.983709] env[75725]: DEBUG oslo_concurrency.lockutils [None req-a681c6b4-3785-4df1-b39e-ba0e96fde057 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75725) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 900.984020] env[75725]: DEBUG oslo_concurrency.lockutils [None req-a681c6b4-3785-4df1-b39e-ba0e96fde057 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75725) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 900.993473] env[75725]: DEBUG oslo_concurrency.lockutils [None req-a681c6b4-3785-4df1-b39e-ba0e96fde057 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75725) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 900.993710] env[75725]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a681c6b4-3785-4df1-b39e-ba0e96fde057 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 900.993875] env[75725]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a681c6b4-3785-4df1-b39e-ba0e96fde057 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=75725) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 902.996796] env[75725]: CRITICAL neutron [None req-a681c6b4-3785-4df1-b39e-ba0e96fde057 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 902.996796] env[75725]: ERROR neutron Traceback (most recent call last): [ 902.996796] env[75725]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 902.996796] env[75725]: ERROR neutron sys.exit(main()) [ 902.996796] env[75725]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 902.996796] env[75725]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 902.996796] env[75725]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 902.996796] env[75725]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 902.996796] env[75725]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 902.996796] env[75725]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 902.996796] env[75725]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 902.996796] env[75725]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 902.996796] env[75725]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 902.996796] env[75725]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 902.996796] env[75725]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 902.996796] env[75725]: ERROR neutron response = self.get(path=path, params=params) [ 902.997585] env[75725]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 902.997585] env[75725]: ERROR neutron raise RuntimeError(msg, last_err) [ 902.997585] env[75725]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 902.997585] env[75725]: ERROR neutron [ 905.577727] env[75957]: INFO neutron.common.config [-] Logging enabled! [ 905.578050] env[75957]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 905.578308] env[75957]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=75957) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 905.578537] env[75957]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 905.578631] env[75957]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 905.579516] env[75957]: WARNING oslo_policy.policy [None req-26768817-83de-4303-af0f-2eac9df03666 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 905.579745] env[75957]: WARNING oslo_policy.policy [None req-26768817-83de-4303-af0f-2eac9df03666 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 905.581047] env[75957]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-26768817-83de-4303-af0f-2eac9df03666 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 905.600942] env[75957]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-26768817-83de-4303-af0f-2eac9df03666 None None] Session token - acquiring [ 905.601808] env[75957]: DEBUG networking_nsxv3.common.locking [None req-26768817-83de-4303-af0f-2eac9df03666 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75957) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 905.602027] env[75957]: DEBUG oslo_concurrency.lockutils [None req-26768817-83de-4303-af0f-2eac9df03666 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75957) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 905.602155] env[75957]: DEBUG oslo_concurrency.lockutils [None req-26768817-83de-4303-af0f-2eac9df03666 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75957) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 905.602529] env[75957]: DEBUG oslo_concurrency.lockutils [None req-26768817-83de-4303-af0f-2eac9df03666 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75957) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 905.613208] env[75957]: DEBUG oslo_concurrency.lockutils [None req-26768817-83de-4303-af0f-2eac9df03666 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75957) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 905.613473] env[75957]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-26768817-83de-4303-af0f-2eac9df03666 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 905.613619] env[75957]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-26768817-83de-4303-af0f-2eac9df03666 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=75957) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 907.619830] env[75957]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-26768817-83de-4303-af0f-2eac9df03666 None None] Session token - acquiring [ 907.620346] env[75957]: DEBUG networking_nsxv3.common.locking [None req-26768817-83de-4303-af0f-2eac9df03666 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75957) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 907.620547] env[75957]: DEBUG oslo_concurrency.lockutils [None req-26768817-83de-4303-af0f-2eac9df03666 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75957) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 907.620676] env[75957]: DEBUG oslo_concurrency.lockutils [None req-26768817-83de-4303-af0f-2eac9df03666 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75957) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 907.620961] env[75957]: DEBUG oslo_concurrency.lockutils [None req-26768817-83de-4303-af0f-2eac9df03666 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75957) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 907.630208] env[75957]: DEBUG oslo_concurrency.lockutils [None req-26768817-83de-4303-af0f-2eac9df03666 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75957) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 907.630661] env[75957]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-26768817-83de-4303-af0f-2eac9df03666 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 907.630912] env[75957]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-26768817-83de-4303-af0f-2eac9df03666 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=75957) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 909.632968] env[75957]: CRITICAL neutron [None req-26768817-83de-4303-af0f-2eac9df03666 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 909.632968] env[75957]: ERROR neutron Traceback (most recent call last): [ 909.632968] env[75957]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 909.632968] env[75957]: ERROR neutron sys.exit(main()) [ 909.632968] env[75957]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 909.632968] env[75957]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 909.632968] env[75957]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 909.632968] env[75957]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 909.632968] env[75957]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 909.632968] env[75957]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 909.632968] env[75957]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 909.632968] env[75957]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 909.632968] env[75957]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 909.632968] env[75957]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 909.632968] env[75957]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 909.632968] env[75957]: ERROR neutron response = self.get(path=path, params=params) [ 909.633698] env[75957]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 909.633698] env[75957]: ERROR neutron raise RuntimeError(msg, last_err) [ 909.633698] env[75957]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 909.633698] env[75957]: ERROR neutron [ 912.249147] env[76024]: INFO neutron.common.config [-] Logging enabled! [ 912.249520] env[76024]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 912.249655] env[76024]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76024) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 912.249870] env[76024]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 912.249992] env[76024]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 912.250866] env[76024]: WARNING oslo_policy.policy [None req-32ca055c-881b-4db8-83ca-0539fdacaa1a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 912.251092] env[76024]: WARNING oslo_policy.policy [None req-32ca055c-881b-4db8-83ca-0539fdacaa1a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 912.252313] env[76024]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-32ca055c-881b-4db8-83ca-0539fdacaa1a None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 912.266478] env[76024]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-32ca055c-881b-4db8-83ca-0539fdacaa1a None None] Session token - acquiring [ 912.267251] env[76024]: DEBUG networking_nsxv3.common.locking [None req-32ca055c-881b-4db8-83ca-0539fdacaa1a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76024) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 912.267439] env[76024]: DEBUG oslo_concurrency.lockutils [None req-32ca055c-881b-4db8-83ca-0539fdacaa1a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76024) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 912.267565] env[76024]: DEBUG oslo_concurrency.lockutils [None req-32ca055c-881b-4db8-83ca-0539fdacaa1a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76024) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 912.267939] env[76024]: DEBUG oslo_concurrency.lockutils [None req-32ca055c-881b-4db8-83ca-0539fdacaa1a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76024) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 912.276976] env[76024]: DEBUG oslo_concurrency.lockutils [None req-32ca055c-881b-4db8-83ca-0539fdacaa1a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76024) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 912.277218] env[76024]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-32ca055c-881b-4db8-83ca-0539fdacaa1a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 912.277367] env[76024]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-32ca055c-881b-4db8-83ca-0539fdacaa1a None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76024) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 914.283434] env[76024]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-32ca055c-881b-4db8-83ca-0539fdacaa1a None None] Session token - acquiring [ 914.283871] env[76024]: DEBUG networking_nsxv3.common.locking [None req-32ca055c-881b-4db8-83ca-0539fdacaa1a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76024) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 914.284064] env[76024]: DEBUG oslo_concurrency.lockutils [None req-32ca055c-881b-4db8-83ca-0539fdacaa1a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76024) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 914.284212] env[76024]: DEBUG oslo_concurrency.lockutils [None req-32ca055c-881b-4db8-83ca-0539fdacaa1a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76024) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 914.284484] env[76024]: DEBUG oslo_concurrency.lockutils [None req-32ca055c-881b-4db8-83ca-0539fdacaa1a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76024) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 914.293828] env[76024]: DEBUG oslo_concurrency.lockutils [None req-32ca055c-881b-4db8-83ca-0539fdacaa1a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76024) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 914.294081] env[76024]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-32ca055c-881b-4db8-83ca-0539fdacaa1a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 914.294218] env[76024]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-32ca055c-881b-4db8-83ca-0539fdacaa1a None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76024) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 916.296903] env[76024]: CRITICAL neutron [None req-32ca055c-881b-4db8-83ca-0539fdacaa1a None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 916.296903] env[76024]: ERROR neutron Traceback (most recent call last): [ 916.296903] env[76024]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 916.296903] env[76024]: ERROR neutron sys.exit(main()) [ 916.296903] env[76024]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 916.296903] env[76024]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 916.296903] env[76024]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 916.296903] env[76024]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 916.296903] env[76024]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 916.296903] env[76024]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 916.296903] env[76024]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 916.296903] env[76024]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 916.296903] env[76024]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 916.296903] env[76024]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 916.296903] env[76024]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 916.296903] env[76024]: ERROR neutron response = self.get(path=path, params=params) [ 916.297545] env[76024]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 916.297545] env[76024]: ERROR neutron raise RuntimeError(msg, last_err) [ 916.297545] env[76024]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 916.297545] env[76024]: ERROR neutron [ 919.063524] env[76093]: INFO neutron.common.config [-] Logging enabled! [ 919.063524] env[76093]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 919.063903] env[76093]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76093) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 919.064158] env[76093]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 919.064229] env[76093]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 919.065060] env[76093]: WARNING oslo_policy.policy [None req-bf6c2fe1-d6f2-4d59-b78f-9d60beb6ecc5 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 919.065412] env[76093]: WARNING oslo_policy.policy [None req-bf6c2fe1-d6f2-4d59-b78f-9d60beb6ecc5 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 919.066503] env[76093]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-bf6c2fe1-d6f2-4d59-b78f-9d60beb6ecc5 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 919.081514] env[76093]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bf6c2fe1-d6f2-4d59-b78f-9d60beb6ecc5 None None] Session token - acquiring [ 919.082274] env[76093]: DEBUG networking_nsxv3.common.locking [None req-bf6c2fe1-d6f2-4d59-b78f-9d60beb6ecc5 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76093) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 919.082619] env[76093]: DEBUG oslo_concurrency.lockutils [None req-bf6c2fe1-d6f2-4d59-b78f-9d60beb6ecc5 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76093) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 919.082709] env[76093]: DEBUG oslo_concurrency.lockutils [None req-bf6c2fe1-d6f2-4d59-b78f-9d60beb6ecc5 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76093) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 919.083193] env[76093]: DEBUG oslo_concurrency.lockutils [None req-bf6c2fe1-d6f2-4d59-b78f-9d60beb6ecc5 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76093) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 919.092577] env[76093]: DEBUG oslo_concurrency.lockutils [None req-bf6c2fe1-d6f2-4d59-b78f-9d60beb6ecc5 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76093) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 919.092906] env[76093]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bf6c2fe1-d6f2-4d59-b78f-9d60beb6ecc5 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 919.093117] env[76093]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bf6c2fe1-d6f2-4d59-b78f-9d60beb6ecc5 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76093) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 921.099330] env[76093]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bf6c2fe1-d6f2-4d59-b78f-9d60beb6ecc5 None None] Session token - acquiring [ 921.099702] env[76093]: DEBUG networking_nsxv3.common.locking [None req-bf6c2fe1-d6f2-4d59-b78f-9d60beb6ecc5 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76093) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 921.099865] env[76093]: DEBUG oslo_concurrency.lockutils [None req-bf6c2fe1-d6f2-4d59-b78f-9d60beb6ecc5 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76093) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 921.100025] env[76093]: DEBUG oslo_concurrency.lockutils [None req-bf6c2fe1-d6f2-4d59-b78f-9d60beb6ecc5 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76093) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 921.100285] env[76093]: DEBUG oslo_concurrency.lockutils [None req-bf6c2fe1-d6f2-4d59-b78f-9d60beb6ecc5 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76093) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 921.109290] env[76093]: DEBUG oslo_concurrency.lockutils [None req-bf6c2fe1-d6f2-4d59-b78f-9d60beb6ecc5 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76093) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 921.109524] env[76093]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bf6c2fe1-d6f2-4d59-b78f-9d60beb6ecc5 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 921.109664] env[76093]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bf6c2fe1-d6f2-4d59-b78f-9d60beb6ecc5 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76093) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 923.112603] env[76093]: CRITICAL neutron [None req-bf6c2fe1-d6f2-4d59-b78f-9d60beb6ecc5 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 923.112603] env[76093]: ERROR neutron Traceback (most recent call last): [ 923.112603] env[76093]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 923.112603] env[76093]: ERROR neutron sys.exit(main()) [ 923.112603] env[76093]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 923.112603] env[76093]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 923.112603] env[76093]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 923.112603] env[76093]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 923.112603] env[76093]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 923.112603] env[76093]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 923.112603] env[76093]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 923.112603] env[76093]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 923.112603] env[76093]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 923.112603] env[76093]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 923.112603] env[76093]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 923.112603] env[76093]: ERROR neutron response = self.get(path=path, params=params) [ 923.113370] env[76093]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 923.113370] env[76093]: ERROR neutron raise RuntimeError(msg, last_err) [ 923.113370] env[76093]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 923.113370] env[76093]: ERROR neutron [ 925.722385] env[76165]: INFO neutron.common.config [-] Logging enabled! [ 925.722714] env[76165]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 925.722857] env[76165]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76165) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 925.723090] env[76165]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 925.723183] env[76165]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 925.724095] env[76165]: WARNING oslo_policy.policy [None req-b8cb2074-4fc0-4c67-8c7c-ab14e7123503 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 925.724325] env[76165]: WARNING oslo_policy.policy [None req-b8cb2074-4fc0-4c67-8c7c-ab14e7123503 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 925.725534] env[76165]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-b8cb2074-4fc0-4c67-8c7c-ab14e7123503 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 925.739473] env[76165]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b8cb2074-4fc0-4c67-8c7c-ab14e7123503 None None] Session token - acquiring [ 925.740240] env[76165]: DEBUG networking_nsxv3.common.locking [None req-b8cb2074-4fc0-4c67-8c7c-ab14e7123503 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76165) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 925.740438] env[76165]: DEBUG oslo_concurrency.lockutils [None req-b8cb2074-4fc0-4c67-8c7c-ab14e7123503 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76165) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 925.740563] env[76165]: DEBUG oslo_concurrency.lockutils [None req-b8cb2074-4fc0-4c67-8c7c-ab14e7123503 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76165) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 925.740919] env[76165]: DEBUG oslo_concurrency.lockutils [None req-b8cb2074-4fc0-4c67-8c7c-ab14e7123503 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76165) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 925.749500] env[76165]: DEBUG oslo_concurrency.lockutils [None req-b8cb2074-4fc0-4c67-8c7c-ab14e7123503 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76165) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 925.749740] env[76165]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b8cb2074-4fc0-4c67-8c7c-ab14e7123503 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 925.749890] env[76165]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b8cb2074-4fc0-4c67-8c7c-ab14e7123503 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76165) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 927.755878] env[76165]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b8cb2074-4fc0-4c67-8c7c-ab14e7123503 None None] Session token - acquiring [ 927.756242] env[76165]: DEBUG networking_nsxv3.common.locking [None req-b8cb2074-4fc0-4c67-8c7c-ab14e7123503 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76165) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 927.756435] env[76165]: DEBUG oslo_concurrency.lockutils [None req-b8cb2074-4fc0-4c67-8c7c-ab14e7123503 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76165) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 927.756562] env[76165]: DEBUG oslo_concurrency.lockutils [None req-b8cb2074-4fc0-4c67-8c7c-ab14e7123503 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76165) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 927.756840] env[76165]: DEBUG oslo_concurrency.lockutils [None req-b8cb2074-4fc0-4c67-8c7c-ab14e7123503 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76165) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 927.765819] env[76165]: DEBUG oslo_concurrency.lockutils [None req-b8cb2074-4fc0-4c67-8c7c-ab14e7123503 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76165) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 927.766117] env[76165]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b8cb2074-4fc0-4c67-8c7c-ab14e7123503 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 927.766268] env[76165]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b8cb2074-4fc0-4c67-8c7c-ab14e7123503 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76165) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 929.768896] env[76165]: CRITICAL neutron [None req-b8cb2074-4fc0-4c67-8c7c-ab14e7123503 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 929.768896] env[76165]: ERROR neutron Traceback (most recent call last): [ 929.768896] env[76165]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 929.768896] env[76165]: ERROR neutron sys.exit(main()) [ 929.768896] env[76165]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 929.768896] env[76165]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 929.768896] env[76165]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 929.768896] env[76165]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 929.768896] env[76165]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 929.768896] env[76165]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 929.768896] env[76165]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 929.768896] env[76165]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 929.768896] env[76165]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 929.768896] env[76165]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 929.768896] env[76165]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 929.768896] env[76165]: ERROR neutron response = self.get(path=path, params=params) [ 929.769482] env[76165]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 929.769482] env[76165]: ERROR neutron raise RuntimeError(msg, last_err) [ 929.769482] env[76165]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 929.769482] env[76165]: ERROR neutron [ 932.217617] env[76182]: INFO neutron.common.config [-] Logging enabled! [ 932.217964] env[76182]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 932.218174] env[76182]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76182) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 932.218408] env[76182]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 932.218501] env[76182]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 932.219348] env[76182]: WARNING oslo_policy.policy [None req-b842de29-8621-45ba-a754-1e9a0b05f229 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 932.219575] env[76182]: WARNING oslo_policy.policy [None req-b842de29-8621-45ba-a754-1e9a0b05f229 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 932.220794] env[76182]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-b842de29-8621-45ba-a754-1e9a0b05f229 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 932.234839] env[76182]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b842de29-8621-45ba-a754-1e9a0b05f229 None None] Session token - acquiring [ 932.235636] env[76182]: DEBUG networking_nsxv3.common.locking [None req-b842de29-8621-45ba-a754-1e9a0b05f229 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76182) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 932.235836] env[76182]: DEBUG oslo_concurrency.lockutils [None req-b842de29-8621-45ba-a754-1e9a0b05f229 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76182) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 932.235961] env[76182]: DEBUG oslo_concurrency.lockutils [None req-b842de29-8621-45ba-a754-1e9a0b05f229 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76182) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 932.236352] env[76182]: DEBUG oslo_concurrency.lockutils [None req-b842de29-8621-45ba-a754-1e9a0b05f229 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76182) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 932.252342] env[76182]: DEBUG oslo_concurrency.lockutils [None req-b842de29-8621-45ba-a754-1e9a0b05f229 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76182) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 932.252605] env[76182]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b842de29-8621-45ba-a754-1e9a0b05f229 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 932.252754] env[76182]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b842de29-8621-45ba-a754-1e9a0b05f229 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76182) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 934.258783] env[76182]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b842de29-8621-45ba-a754-1e9a0b05f229 None None] Session token - acquiring [ 934.259327] env[76182]: DEBUG networking_nsxv3.common.locking [None req-b842de29-8621-45ba-a754-1e9a0b05f229 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76182) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 934.259542] env[76182]: DEBUG oslo_concurrency.lockutils [None req-b842de29-8621-45ba-a754-1e9a0b05f229 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76182) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 934.259708] env[76182]: DEBUG oslo_concurrency.lockutils [None req-b842de29-8621-45ba-a754-1e9a0b05f229 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76182) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 934.259994] env[76182]: DEBUG oslo_concurrency.lockutils [None req-b842de29-8621-45ba-a754-1e9a0b05f229 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76182) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 934.269258] env[76182]: DEBUG oslo_concurrency.lockutils [None req-b842de29-8621-45ba-a754-1e9a0b05f229 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76182) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 934.269504] env[76182]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b842de29-8621-45ba-a754-1e9a0b05f229 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 934.269645] env[76182]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b842de29-8621-45ba-a754-1e9a0b05f229 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76182) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 936.272594] env[76182]: CRITICAL neutron [None req-b842de29-8621-45ba-a754-1e9a0b05f229 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 936.272594] env[76182]: ERROR neutron Traceback (most recent call last): [ 936.272594] env[76182]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 936.272594] env[76182]: ERROR neutron sys.exit(main()) [ 936.272594] env[76182]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 936.272594] env[76182]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 936.272594] env[76182]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 936.272594] env[76182]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 936.272594] env[76182]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 936.272594] env[76182]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 936.272594] env[76182]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 936.272594] env[76182]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 936.272594] env[76182]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 936.272594] env[76182]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 936.272594] env[76182]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 936.272594] env[76182]: ERROR neutron response = self.get(path=path, params=params) [ 936.273333] env[76182]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 936.273333] env[76182]: ERROR neutron raise RuntimeError(msg, last_err) [ 936.273333] env[76182]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 936.273333] env[76182]: ERROR neutron [ 938.725184] env[76243]: INFO neutron.common.config [-] Logging enabled! [ 938.725184] env[76243]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 938.725184] env[76243]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76243) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 938.725184] env[76243]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 938.725184] env[76243]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 938.725184] env[76243]: WARNING oslo_policy.policy [None req-a6ebf6e7-62d3-4189-8233-6ff943c63f83 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 938.725184] env[76243]: WARNING oslo_policy.policy [None req-a6ebf6e7-62d3-4189-8233-6ff943c63f83 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 938.725690] env[76243]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-a6ebf6e7-62d3-4189-8233-6ff943c63f83 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 938.739682] env[76243]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a6ebf6e7-62d3-4189-8233-6ff943c63f83 None None] Session token - acquiring [ 938.740516] env[76243]: DEBUG networking_nsxv3.common.locking [None req-a6ebf6e7-62d3-4189-8233-6ff943c63f83 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76243) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 938.740728] env[76243]: DEBUG oslo_concurrency.lockutils [None req-a6ebf6e7-62d3-4189-8233-6ff943c63f83 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76243) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 938.740857] env[76243]: DEBUG oslo_concurrency.lockutils [None req-a6ebf6e7-62d3-4189-8233-6ff943c63f83 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76243) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 938.741222] env[76243]: DEBUG oslo_concurrency.lockutils [None req-a6ebf6e7-62d3-4189-8233-6ff943c63f83 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76243) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 938.750238] env[76243]: DEBUG oslo_concurrency.lockutils [None req-a6ebf6e7-62d3-4189-8233-6ff943c63f83 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76243) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 938.750489] env[76243]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a6ebf6e7-62d3-4189-8233-6ff943c63f83 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 938.750638] env[76243]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a6ebf6e7-62d3-4189-8233-6ff943c63f83 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76243) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 940.756909] env[76243]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a6ebf6e7-62d3-4189-8233-6ff943c63f83 None None] Session token - acquiring [ 940.756909] env[76243]: DEBUG networking_nsxv3.common.locking [None req-a6ebf6e7-62d3-4189-8233-6ff943c63f83 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76243) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 940.757361] env[76243]: DEBUG oslo_concurrency.lockutils [None req-a6ebf6e7-62d3-4189-8233-6ff943c63f83 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76243) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 940.757361] env[76243]: DEBUG oslo_concurrency.lockutils [None req-a6ebf6e7-62d3-4189-8233-6ff943c63f83 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76243) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 940.757457] env[76243]: DEBUG oslo_concurrency.lockutils [None req-a6ebf6e7-62d3-4189-8233-6ff943c63f83 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76243) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 940.769841] env[76243]: DEBUG oslo_concurrency.lockutils [None req-a6ebf6e7-62d3-4189-8233-6ff943c63f83 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76243) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 940.771122] env[76243]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a6ebf6e7-62d3-4189-8233-6ff943c63f83 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 940.771309] env[76243]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a6ebf6e7-62d3-4189-8233-6ff943c63f83 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76243) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 942.772926] env[76243]: CRITICAL neutron [None req-a6ebf6e7-62d3-4189-8233-6ff943c63f83 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 942.772926] env[76243]: ERROR neutron Traceback (most recent call last): [ 942.772926] env[76243]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 942.772926] env[76243]: ERROR neutron sys.exit(main()) [ 942.772926] env[76243]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 942.772926] env[76243]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 942.772926] env[76243]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 942.772926] env[76243]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 942.772926] env[76243]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 942.772926] env[76243]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 942.772926] env[76243]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 942.772926] env[76243]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 942.772926] env[76243]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 942.772926] env[76243]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 942.772926] env[76243]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 942.772926] env[76243]: ERROR neutron response = self.get(path=path, params=params) [ 942.773645] env[76243]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 942.773645] env[76243]: ERROR neutron raise RuntimeError(msg, last_err) [ 942.773645] env[76243]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 942.773645] env[76243]: ERROR neutron [ 945.379277] env[76356]: INFO neutron.common.config [-] Logging enabled! [ 945.379557] env[76356]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 945.379795] env[76356]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76356) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 945.380044] env[76356]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 945.380147] env[76356]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 945.380985] env[76356]: WARNING oslo_policy.policy [None req-1c2acea9-9f8b-421c-bd8e-97c72a7aed6d None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 945.381202] env[76356]: WARNING oslo_policy.policy [None req-1c2acea9-9f8b-421c-bd8e-97c72a7aed6d None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 945.382447] env[76356]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-1c2acea9-9f8b-421c-bd8e-97c72a7aed6d None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 945.396798] env[76356]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1c2acea9-9f8b-421c-bd8e-97c72a7aed6d None None] Session token - acquiring [ 945.397553] env[76356]: DEBUG networking_nsxv3.common.locking [None req-1c2acea9-9f8b-421c-bd8e-97c72a7aed6d None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76356) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 945.397751] env[76356]: DEBUG oslo_concurrency.lockutils [None req-1c2acea9-9f8b-421c-bd8e-97c72a7aed6d None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76356) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 945.397878] env[76356]: DEBUG oslo_concurrency.lockutils [None req-1c2acea9-9f8b-421c-bd8e-97c72a7aed6d None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76356) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 945.398232] env[76356]: DEBUG oslo_concurrency.lockutils [None req-1c2acea9-9f8b-421c-bd8e-97c72a7aed6d None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76356) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 945.406992] env[76356]: DEBUG oslo_concurrency.lockutils [None req-1c2acea9-9f8b-421c-bd8e-97c72a7aed6d None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76356) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 945.407225] env[76356]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1c2acea9-9f8b-421c-bd8e-97c72a7aed6d None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 945.407367] env[76356]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1c2acea9-9f8b-421c-bd8e-97c72a7aed6d None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76356) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 947.413633] env[76356]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1c2acea9-9f8b-421c-bd8e-97c72a7aed6d None None] Session token - acquiring [ 947.414294] env[76356]: DEBUG networking_nsxv3.common.locking [None req-1c2acea9-9f8b-421c-bd8e-97c72a7aed6d None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76356) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 947.414481] env[76356]: DEBUG oslo_concurrency.lockutils [None req-1c2acea9-9f8b-421c-bd8e-97c72a7aed6d None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76356) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 947.414602] env[76356]: DEBUG oslo_concurrency.lockutils [None req-1c2acea9-9f8b-421c-bd8e-97c72a7aed6d None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76356) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 947.414958] env[76356]: DEBUG oslo_concurrency.lockutils [None req-1c2acea9-9f8b-421c-bd8e-97c72a7aed6d None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76356) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 947.424991] env[76356]: DEBUG oslo_concurrency.lockutils [None req-1c2acea9-9f8b-421c-bd8e-97c72a7aed6d None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76356) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 947.425398] env[76356]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1c2acea9-9f8b-421c-bd8e-97c72a7aed6d None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 947.425628] env[76356]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1c2acea9-9f8b-421c-bd8e-97c72a7aed6d None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76356) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 949.428359] env[76356]: CRITICAL neutron [None req-1c2acea9-9f8b-421c-bd8e-97c72a7aed6d None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 949.428359] env[76356]: ERROR neutron Traceback (most recent call last): [ 949.428359] env[76356]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 949.428359] env[76356]: ERROR neutron sys.exit(main()) [ 949.428359] env[76356]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 949.428359] env[76356]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 949.428359] env[76356]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 949.428359] env[76356]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 949.428359] env[76356]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 949.428359] env[76356]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 949.428359] env[76356]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 949.428359] env[76356]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 949.428359] env[76356]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 949.428359] env[76356]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 949.428359] env[76356]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 949.428359] env[76356]: ERROR neutron response = self.get(path=path, params=params) [ 949.428998] env[76356]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 949.428998] env[76356]: ERROR neutron raise RuntimeError(msg, last_err) [ 949.428998] env[76356]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 949.428998] env[76356]: ERROR neutron [ 952.079680] env[76578]: INFO neutron.common.config [-] Logging enabled! [ 952.079920] env[76578]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 952.080233] env[76578]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76578) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 952.080486] env[76578]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 952.080550] env[76578]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 952.081417] env[76578]: WARNING oslo_policy.policy [None req-bf18027e-e71b-4738-bdc5-deaee6e6d875 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 952.081651] env[76578]: WARNING oslo_policy.policy [None req-bf18027e-e71b-4738-bdc5-deaee6e6d875 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 952.082908] env[76578]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-bf18027e-e71b-4738-bdc5-deaee6e6d875 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 952.098521] env[76578]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bf18027e-e71b-4738-bdc5-deaee6e6d875 None None] Session token - acquiring [ 952.099384] env[76578]: DEBUG networking_nsxv3.common.locking [None req-bf18027e-e71b-4738-bdc5-deaee6e6d875 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76578) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 952.099566] env[76578]: DEBUG oslo_concurrency.lockutils [None req-bf18027e-e71b-4738-bdc5-deaee6e6d875 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76578) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 952.099696] env[76578]: DEBUG oslo_concurrency.lockutils [None req-bf18027e-e71b-4738-bdc5-deaee6e6d875 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76578) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 952.100083] env[76578]: DEBUG oslo_concurrency.lockutils [None req-bf18027e-e71b-4738-bdc5-deaee6e6d875 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76578) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 952.109059] env[76578]: DEBUG oslo_concurrency.lockutils [None req-bf18027e-e71b-4738-bdc5-deaee6e6d875 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76578) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 952.109322] env[76578]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bf18027e-e71b-4738-bdc5-deaee6e6d875 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 952.109473] env[76578]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bf18027e-e71b-4738-bdc5-deaee6e6d875 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76578) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 954.115615] env[76578]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bf18027e-e71b-4738-bdc5-deaee6e6d875 None None] Session token - acquiring [ 954.116046] env[76578]: DEBUG networking_nsxv3.common.locking [None req-bf18027e-e71b-4738-bdc5-deaee6e6d875 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76578) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 954.116257] env[76578]: DEBUG oslo_concurrency.lockutils [None req-bf18027e-e71b-4738-bdc5-deaee6e6d875 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76578) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 954.116379] env[76578]: DEBUG oslo_concurrency.lockutils [None req-bf18027e-e71b-4738-bdc5-deaee6e6d875 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76578) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 954.116647] env[76578]: DEBUG oslo_concurrency.lockutils [None req-bf18027e-e71b-4738-bdc5-deaee6e6d875 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76578) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 954.125839] env[76578]: DEBUG oslo_concurrency.lockutils [None req-bf18027e-e71b-4738-bdc5-deaee6e6d875 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76578) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 954.126081] env[76578]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bf18027e-e71b-4738-bdc5-deaee6e6d875 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 954.126221] env[76578]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bf18027e-e71b-4738-bdc5-deaee6e6d875 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76578) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 956.128910] env[76578]: CRITICAL neutron [None req-bf18027e-e71b-4738-bdc5-deaee6e6d875 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 956.128910] env[76578]: ERROR neutron Traceback (most recent call last): [ 956.128910] env[76578]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 956.128910] env[76578]: ERROR neutron sys.exit(main()) [ 956.128910] env[76578]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 956.128910] env[76578]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 956.128910] env[76578]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 956.128910] env[76578]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 956.128910] env[76578]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 956.128910] env[76578]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 956.128910] env[76578]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 956.128910] env[76578]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 956.128910] env[76578]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 956.128910] env[76578]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 956.128910] env[76578]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 956.128910] env[76578]: ERROR neutron response = self.get(path=path, params=params) [ 956.129633] env[76578]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 956.129633] env[76578]: ERROR neutron raise RuntimeError(msg, last_err) [ 956.129633] env[76578]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 956.129633] env[76578]: ERROR neutron [ 958.885724] env[76849]: INFO neutron.common.config [-] Logging enabled! [ 958.886001] env[76849]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 958.886298] env[76849]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76849) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 958.886541] env[76849]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 958.886632] env[76849]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 958.888412] env[76849]: WARNING oslo_policy.policy [None req-aad07ca4-2b87-49ac-9674-eafdfe675c02 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 958.888412] env[76849]: WARNING oslo_policy.policy [None req-aad07ca4-2b87-49ac-9674-eafdfe675c02 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 958.889227] env[76849]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-aad07ca4-2b87-49ac-9674-eafdfe675c02 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 958.904323] env[76849]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-aad07ca4-2b87-49ac-9674-eafdfe675c02 None None] Session token - acquiring [ 958.905189] env[76849]: DEBUG networking_nsxv3.common.locking [None req-aad07ca4-2b87-49ac-9674-eafdfe675c02 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76849) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 958.905649] env[76849]: DEBUG oslo_concurrency.lockutils [None req-aad07ca4-2b87-49ac-9674-eafdfe675c02 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76849) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 958.905816] env[76849]: DEBUG oslo_concurrency.lockutils [None req-aad07ca4-2b87-49ac-9674-eafdfe675c02 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76849) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 958.906252] env[76849]: DEBUG oslo_concurrency.lockutils [None req-aad07ca4-2b87-49ac-9674-eafdfe675c02 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76849) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 958.915721] env[76849]: DEBUG oslo_concurrency.lockutils [None req-aad07ca4-2b87-49ac-9674-eafdfe675c02 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76849) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 958.915983] env[76849]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-aad07ca4-2b87-49ac-9674-eafdfe675c02 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 958.916152] env[76849]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-aad07ca4-2b87-49ac-9674-eafdfe675c02 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76849) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 960.922879] env[76849]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-aad07ca4-2b87-49ac-9674-eafdfe675c02 None None] Session token - acquiring [ 960.922879] env[76849]: DEBUG networking_nsxv3.common.locking [None req-aad07ca4-2b87-49ac-9674-eafdfe675c02 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76849) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 960.922879] env[76849]: DEBUG oslo_concurrency.lockutils [None req-aad07ca4-2b87-49ac-9674-eafdfe675c02 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76849) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 960.923319] env[76849]: DEBUG oslo_concurrency.lockutils [None req-aad07ca4-2b87-49ac-9674-eafdfe675c02 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76849) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 960.923319] env[76849]: DEBUG oslo_concurrency.lockutils [None req-aad07ca4-2b87-49ac-9674-eafdfe675c02 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76849) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 960.932634] env[76849]: DEBUG oslo_concurrency.lockutils [None req-aad07ca4-2b87-49ac-9674-eafdfe675c02 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76849) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 960.932884] env[76849]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-aad07ca4-2b87-49ac-9674-eafdfe675c02 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 960.933024] env[76849]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-aad07ca4-2b87-49ac-9674-eafdfe675c02 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76849) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 962.936053] env[76849]: CRITICAL neutron [None req-aad07ca4-2b87-49ac-9674-eafdfe675c02 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 962.936053] env[76849]: ERROR neutron Traceback (most recent call last): [ 962.936053] env[76849]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 962.936053] env[76849]: ERROR neutron sys.exit(main()) [ 962.936053] env[76849]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 962.936053] env[76849]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 962.936053] env[76849]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 962.936053] env[76849]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 962.936053] env[76849]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 962.936053] env[76849]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 962.936053] env[76849]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 962.936053] env[76849]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 962.936053] env[76849]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 962.936053] env[76849]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 962.936053] env[76849]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 962.936053] env[76849]: ERROR neutron response = self.get(path=path, params=params) [ 962.936904] env[76849]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 962.936904] env[76849]: ERROR neutron raise RuntimeError(msg, last_err) [ 962.936904] env[76849]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 962.936904] env[76849]: ERROR neutron [ 965.553425] env[77071]: INFO neutron.common.config [-] Logging enabled! [ 965.553745] env[77071]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 965.553980] env[77071]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77071) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 965.554219] env[77071]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 965.554314] env[77071]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 965.555172] env[77071]: WARNING oslo_policy.policy [None req-a9065282-9254-4ba6-8f9d-823c4d01ccce None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 965.555389] env[77071]: WARNING oslo_policy.policy [None req-a9065282-9254-4ba6-8f9d-823c4d01ccce None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 965.556621] env[77071]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-a9065282-9254-4ba6-8f9d-823c4d01ccce None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 965.570791] env[77071]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a9065282-9254-4ba6-8f9d-823c4d01ccce None None] Session token - acquiring [ 965.571573] env[77071]: DEBUG networking_nsxv3.common.locking [None req-a9065282-9254-4ba6-8f9d-823c4d01ccce None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77071) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 965.571777] env[77071]: DEBUG oslo_concurrency.lockutils [None req-a9065282-9254-4ba6-8f9d-823c4d01ccce None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77071) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 965.571929] env[77071]: DEBUG oslo_concurrency.lockutils [None req-a9065282-9254-4ba6-8f9d-823c4d01ccce None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77071) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 965.572312] env[77071]: DEBUG oslo_concurrency.lockutils [None req-a9065282-9254-4ba6-8f9d-823c4d01ccce None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77071) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 965.585728] env[77071]: DEBUG oslo_concurrency.lockutils [None req-a9065282-9254-4ba6-8f9d-823c4d01ccce None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77071) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 965.585728] env[77071]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a9065282-9254-4ba6-8f9d-823c4d01ccce None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 965.585728] env[77071]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a9065282-9254-4ba6-8f9d-823c4d01ccce None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77071) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 967.588917] env[77071]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a9065282-9254-4ba6-8f9d-823c4d01ccce None None] Session token - acquiring [ 967.589216] env[77071]: DEBUG networking_nsxv3.common.locking [None req-a9065282-9254-4ba6-8f9d-823c4d01ccce None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77071) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 967.589374] env[77071]: DEBUG oslo_concurrency.lockutils [None req-a9065282-9254-4ba6-8f9d-823c4d01ccce None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77071) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 967.589493] env[77071]: DEBUG oslo_concurrency.lockutils [None req-a9065282-9254-4ba6-8f9d-823c4d01ccce None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77071) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 967.589758] env[77071]: DEBUG oslo_concurrency.lockutils [None req-a9065282-9254-4ba6-8f9d-823c4d01ccce None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77071) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 967.598918] env[77071]: DEBUG oslo_concurrency.lockutils [None req-a9065282-9254-4ba6-8f9d-823c4d01ccce None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77071) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 967.599169] env[77071]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a9065282-9254-4ba6-8f9d-823c4d01ccce None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 967.599303] env[77071]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a9065282-9254-4ba6-8f9d-823c4d01ccce None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77071) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 969.600996] env[77071]: CRITICAL neutron [None req-a9065282-9254-4ba6-8f9d-823c4d01ccce None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 969.600996] env[77071]: ERROR neutron Traceback (most recent call last): [ 969.600996] env[77071]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 969.600996] env[77071]: ERROR neutron sys.exit(main()) [ 969.600996] env[77071]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 969.600996] env[77071]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 969.600996] env[77071]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 969.600996] env[77071]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 969.600996] env[77071]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 969.600996] env[77071]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 969.600996] env[77071]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 969.600996] env[77071]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 969.600996] env[77071]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 969.600996] env[77071]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 969.600996] env[77071]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 969.600996] env[77071]: ERROR neutron response = self.get(path=path, params=params) [ 969.602428] env[77071]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 969.602428] env[77071]: ERROR neutron raise RuntimeError(msg, last_err) [ 969.602428] env[77071]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 969.602428] env[77071]: ERROR neutron [ 972.220386] env[77184]: INFO neutron.common.config [-] Logging enabled! [ 972.220699] env[77184]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 972.220970] env[77184]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77184) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 972.221207] env[77184]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 972.221304] env[77184]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 972.222171] env[77184]: WARNING oslo_policy.policy [None req-95dff04f-436c-40f8-aa3e-07e873946194 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 972.222391] env[77184]: WARNING oslo_policy.policy [None req-95dff04f-436c-40f8-aa3e-07e873946194 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 972.223639] env[77184]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-95dff04f-436c-40f8-aa3e-07e873946194 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 972.238175] env[77184]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-95dff04f-436c-40f8-aa3e-07e873946194 None None] Session token - acquiring [ 972.239005] env[77184]: DEBUG networking_nsxv3.common.locking [None req-95dff04f-436c-40f8-aa3e-07e873946194 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77184) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 972.239212] env[77184]: DEBUG oslo_concurrency.lockutils [None req-95dff04f-436c-40f8-aa3e-07e873946194 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77184) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 972.239339] env[77184]: DEBUG oslo_concurrency.lockutils [None req-95dff04f-436c-40f8-aa3e-07e873946194 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77184) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 972.239700] env[77184]: DEBUG oslo_concurrency.lockutils [None req-95dff04f-436c-40f8-aa3e-07e873946194 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77184) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 972.248923] env[77184]: DEBUG oslo_concurrency.lockutils [None req-95dff04f-436c-40f8-aa3e-07e873946194 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77184) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 972.249183] env[77184]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-95dff04f-436c-40f8-aa3e-07e873946194 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 972.249334] env[77184]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-95dff04f-436c-40f8-aa3e-07e873946194 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77184) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 974.256239] env[77184]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-95dff04f-436c-40f8-aa3e-07e873946194 None None] Session token - acquiring [ 974.256547] env[77184]: DEBUG networking_nsxv3.common.locking [None req-95dff04f-436c-40f8-aa3e-07e873946194 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77184) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 974.256618] env[77184]: DEBUG oslo_concurrency.lockutils [None req-95dff04f-436c-40f8-aa3e-07e873946194 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77184) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 974.256691] env[77184]: DEBUG oslo_concurrency.lockutils [None req-95dff04f-436c-40f8-aa3e-07e873946194 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77184) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 974.256976] env[77184]: DEBUG oslo_concurrency.lockutils [None req-95dff04f-436c-40f8-aa3e-07e873946194 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77184) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 974.265807] env[77184]: DEBUG oslo_concurrency.lockutils [None req-95dff04f-436c-40f8-aa3e-07e873946194 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77184) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 974.266036] env[77184]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-95dff04f-436c-40f8-aa3e-07e873946194 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 974.266175] env[77184]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-95dff04f-436c-40f8-aa3e-07e873946194 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77184) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 976.268859] env[77184]: CRITICAL neutron [None req-95dff04f-436c-40f8-aa3e-07e873946194 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 976.268859] env[77184]: ERROR neutron Traceback (most recent call last): [ 976.268859] env[77184]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 976.268859] env[77184]: ERROR neutron sys.exit(main()) [ 976.268859] env[77184]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 976.268859] env[77184]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 976.268859] env[77184]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 976.268859] env[77184]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 976.268859] env[77184]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 976.268859] env[77184]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 976.268859] env[77184]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 976.268859] env[77184]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 976.268859] env[77184]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 976.268859] env[77184]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 976.268859] env[77184]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 976.268859] env[77184]: ERROR neutron response = self.get(path=path, params=params) [ 976.269508] env[77184]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 976.269508] env[77184]: ERROR neutron raise RuntimeError(msg, last_err) [ 976.269508] env[77184]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 976.269508] env[77184]: ERROR neutron [ 978.747611] env[77264]: INFO neutron.common.config [-] Logging enabled! [ 978.747935] env[77264]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 978.748164] env[77264]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77264) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 978.748395] env[77264]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 978.748491] env[77264]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 978.749331] env[77264]: WARNING oslo_policy.policy [None req-830aa009-3536-4876-bc1c-aa962062619f None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 978.749554] env[77264]: WARNING oslo_policy.policy [None req-830aa009-3536-4876-bc1c-aa962062619f None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 978.750842] env[77264]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-830aa009-3536-4876-bc1c-aa962062619f None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 978.764947] env[77264]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-830aa009-3536-4876-bc1c-aa962062619f None None] Session token - acquiring [ 978.765734] env[77264]: DEBUG networking_nsxv3.common.locking [None req-830aa009-3536-4876-bc1c-aa962062619f None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77264) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 978.765967] env[77264]: DEBUG oslo_concurrency.lockutils [None req-830aa009-3536-4876-bc1c-aa962062619f None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77264) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 978.766080] env[77264]: DEBUG oslo_concurrency.lockutils [None req-830aa009-3536-4876-bc1c-aa962062619f None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77264) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 978.772217] env[77264]: DEBUG oslo_concurrency.lockutils [None req-830aa009-3536-4876-bc1c-aa962062619f None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77264) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 978.775507] env[77264]: DEBUG oslo_concurrency.lockutils [None req-830aa009-3536-4876-bc1c-aa962062619f None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77264) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 978.775755] env[77264]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-830aa009-3536-4876-bc1c-aa962062619f None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 978.775903] env[77264]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-830aa009-3536-4876-bc1c-aa962062619f None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77264) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 980.782064] env[77264]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-830aa009-3536-4876-bc1c-aa962062619f None None] Session token - acquiring [ 980.782363] env[77264]: DEBUG networking_nsxv3.common.locking [None req-830aa009-3536-4876-bc1c-aa962062619f None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77264) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 980.782555] env[77264]: DEBUG oslo_concurrency.lockutils [None req-830aa009-3536-4876-bc1c-aa962062619f None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77264) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 980.782678] env[77264]: DEBUG oslo_concurrency.lockutils [None req-830aa009-3536-4876-bc1c-aa962062619f None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77264) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 980.782955] env[77264]: DEBUG oslo_concurrency.lockutils [None req-830aa009-3536-4876-bc1c-aa962062619f None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77264) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 980.792197] env[77264]: DEBUG oslo_concurrency.lockutils [None req-830aa009-3536-4876-bc1c-aa962062619f None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77264) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 980.792433] env[77264]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-830aa009-3536-4876-bc1c-aa962062619f None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 980.792574] env[77264]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-830aa009-3536-4876-bc1c-aa962062619f None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77264) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 982.795458] env[77264]: CRITICAL neutron [None req-830aa009-3536-4876-bc1c-aa962062619f None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 982.795458] env[77264]: ERROR neutron Traceback (most recent call last): [ 982.795458] env[77264]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 982.795458] env[77264]: ERROR neutron sys.exit(main()) [ 982.795458] env[77264]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 982.795458] env[77264]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 982.795458] env[77264]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 982.795458] env[77264]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 982.795458] env[77264]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 982.795458] env[77264]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 982.795458] env[77264]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 982.795458] env[77264]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 982.795458] env[77264]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 982.795458] env[77264]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 982.795458] env[77264]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 982.795458] env[77264]: ERROR neutron response = self.get(path=path, params=params) [ 982.796521] env[77264]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 982.796521] env[77264]: ERROR neutron raise RuntimeError(msg, last_err) [ 982.796521] env[77264]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 982.796521] env[77264]: ERROR neutron [ 985.183263] env[77358]: INFO neutron.common.config [-] Logging enabled! [ 985.183581] env[77358]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 985.183777] env[77358]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77358) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 985.184045] env[77358]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 985.184137] env[77358]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 985.185007] env[77358]: WARNING oslo_policy.policy [None req-0812f3b9-0962-45a8-8baf-bfd6a82043f3 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 985.185228] env[77358]: WARNING oslo_policy.policy [None req-0812f3b9-0962-45a8-8baf-bfd6a82043f3 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 985.186449] env[77358]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-0812f3b9-0962-45a8-8baf-bfd6a82043f3 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 985.199303] env[77358]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0812f3b9-0962-45a8-8baf-bfd6a82043f3 None None] Session token - acquiring [ 985.200120] env[77358]: DEBUG networking_nsxv3.common.locking [None req-0812f3b9-0962-45a8-8baf-bfd6a82043f3 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77358) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 985.200325] env[77358]: DEBUG oslo_concurrency.lockutils [None req-0812f3b9-0962-45a8-8baf-bfd6a82043f3 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77358) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 985.200451] env[77358]: DEBUG oslo_concurrency.lockutils [None req-0812f3b9-0962-45a8-8baf-bfd6a82043f3 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77358) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 985.200809] env[77358]: DEBUG oslo_concurrency.lockutils [None req-0812f3b9-0962-45a8-8baf-bfd6a82043f3 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77358) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 985.210181] env[77358]: DEBUG oslo_concurrency.lockutils [None req-0812f3b9-0962-45a8-8baf-bfd6a82043f3 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77358) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 985.210428] env[77358]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0812f3b9-0962-45a8-8baf-bfd6a82043f3 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 985.210574] env[77358]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0812f3b9-0962-45a8-8baf-bfd6a82043f3 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77358) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 987.217913] env[77358]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0812f3b9-0962-45a8-8baf-bfd6a82043f3 None None] Session token - acquiring [ 987.218201] env[77358]: DEBUG networking_nsxv3.common.locking [None req-0812f3b9-0962-45a8-8baf-bfd6a82043f3 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77358) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 987.218413] env[77358]: DEBUG oslo_concurrency.lockutils [None req-0812f3b9-0962-45a8-8baf-bfd6a82043f3 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77358) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 987.218542] env[77358]: DEBUG oslo_concurrency.lockutils [None req-0812f3b9-0962-45a8-8baf-bfd6a82043f3 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77358) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 987.218812] env[77358]: DEBUG oslo_concurrency.lockutils [None req-0812f3b9-0962-45a8-8baf-bfd6a82043f3 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77358) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 987.227790] env[77358]: DEBUG oslo_concurrency.lockutils [None req-0812f3b9-0962-45a8-8baf-bfd6a82043f3 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77358) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 987.228063] env[77358]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0812f3b9-0962-45a8-8baf-bfd6a82043f3 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 987.228216] env[77358]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0812f3b9-0962-45a8-8baf-bfd6a82043f3 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77358) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 989.231002] env[77358]: CRITICAL neutron [None req-0812f3b9-0962-45a8-8baf-bfd6a82043f3 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 989.231002] env[77358]: ERROR neutron Traceback (most recent call last): [ 989.231002] env[77358]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 989.231002] env[77358]: ERROR neutron sys.exit(main()) [ 989.231002] env[77358]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 989.231002] env[77358]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 989.231002] env[77358]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 989.231002] env[77358]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 989.231002] env[77358]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 989.231002] env[77358]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 989.231002] env[77358]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 989.231002] env[77358]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 989.231002] env[77358]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 989.231002] env[77358]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 989.231002] env[77358]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 989.231002] env[77358]: ERROR neutron response = self.get(path=path, params=params) [ 989.231811] env[77358]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 989.231811] env[77358]: ERROR neutron raise RuntimeError(msg, last_err) [ 989.231811] env[77358]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 989.231811] env[77358]: ERROR neutron [ 991.740529] env[77446]: INFO neutron.common.config [-] Logging enabled! [ 991.740918] env[77446]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 991.741054] env[77446]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77446) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 991.741283] env[77446]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 991.741379] env[77446]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 991.742239] env[77446]: WARNING oslo_policy.policy [None req-81201f20-0e6c-43bf-80e1-dd1257b412b4 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 991.742458] env[77446]: WARNING oslo_policy.policy [None req-81201f20-0e6c-43bf-80e1-dd1257b412b4 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 991.743726] env[77446]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-81201f20-0e6c-43bf-80e1-dd1257b412b4 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 991.758104] env[77446]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-81201f20-0e6c-43bf-80e1-dd1257b412b4 None None] Session token - acquiring [ 991.758905] env[77446]: DEBUG networking_nsxv3.common.locking [None req-81201f20-0e6c-43bf-80e1-dd1257b412b4 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77446) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 991.759110] env[77446]: DEBUG oslo_concurrency.lockutils [None req-81201f20-0e6c-43bf-80e1-dd1257b412b4 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77446) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 991.759240] env[77446]: DEBUG oslo_concurrency.lockutils [None req-81201f20-0e6c-43bf-80e1-dd1257b412b4 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77446) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 991.759605] env[77446]: DEBUG oslo_concurrency.lockutils [None req-81201f20-0e6c-43bf-80e1-dd1257b412b4 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77446) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 991.768610] env[77446]: DEBUG oslo_concurrency.lockutils [None req-81201f20-0e6c-43bf-80e1-dd1257b412b4 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77446) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 991.768872] env[77446]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-81201f20-0e6c-43bf-80e1-dd1257b412b4 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 991.769028] env[77446]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-81201f20-0e6c-43bf-80e1-dd1257b412b4 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77446) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 993.775140] env[77446]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-81201f20-0e6c-43bf-80e1-dd1257b412b4 None None] Session token - acquiring [ 993.775385] env[77446]: DEBUG networking_nsxv3.common.locking [None req-81201f20-0e6c-43bf-80e1-dd1257b412b4 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77446) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 993.775895] env[77446]: DEBUG oslo_concurrency.lockutils [None req-81201f20-0e6c-43bf-80e1-dd1257b412b4 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77446) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 993.775895] env[77446]: DEBUG oslo_concurrency.lockutils [None req-81201f20-0e6c-43bf-80e1-dd1257b412b4 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77446) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 993.775895] env[77446]: DEBUG oslo_concurrency.lockutils [None req-81201f20-0e6c-43bf-80e1-dd1257b412b4 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77446) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 993.785246] env[77446]: DEBUG oslo_concurrency.lockutils [None req-81201f20-0e6c-43bf-80e1-dd1257b412b4 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77446) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 993.785547] env[77446]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-81201f20-0e6c-43bf-80e1-dd1257b412b4 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 993.785703] env[77446]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-81201f20-0e6c-43bf-80e1-dd1257b412b4 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77446) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 995.788999] env[77446]: CRITICAL neutron [None req-81201f20-0e6c-43bf-80e1-dd1257b412b4 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 995.788999] env[77446]: ERROR neutron Traceback (most recent call last): [ 995.788999] env[77446]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 995.788999] env[77446]: ERROR neutron sys.exit(main()) [ 995.788999] env[77446]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 995.788999] env[77446]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 995.788999] env[77446]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 995.788999] env[77446]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 995.788999] env[77446]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 995.788999] env[77446]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 995.788999] env[77446]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 995.788999] env[77446]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 995.788999] env[77446]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 995.788999] env[77446]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 995.788999] env[77446]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 995.788999] env[77446]: ERROR neutron response = self.get(path=path, params=params) [ 995.790034] env[77446]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 995.790034] env[77446]: ERROR neutron raise RuntimeError(msg, last_err) [ 995.790034] env[77446]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 995.790034] env[77446]: ERROR neutron [ 998.256671] env[77456]: INFO neutron.common.config [-] Logging enabled! [ 998.257048] env[77456]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 998.257167] env[77456]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77456) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 998.257436] env[77456]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 998.257491] env[77456]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 998.258280] env[77456]: WARNING oslo_policy.policy [None req-96ee2dac-0a0f-4a64-9ce8-2aeff5240980 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 998.258499] env[77456]: WARNING oslo_policy.policy [None req-96ee2dac-0a0f-4a64-9ce8-2aeff5240980 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 998.259681] env[77456]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-96ee2dac-0a0f-4a64-9ce8-2aeff5240980 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 998.277379] env[77456]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-96ee2dac-0a0f-4a64-9ce8-2aeff5240980 None None] Session token - acquiring [ 998.278547] env[77456]: DEBUG networking_nsxv3.common.locking [None req-96ee2dac-0a0f-4a64-9ce8-2aeff5240980 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77456) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 998.278757] env[77456]: DEBUG oslo_concurrency.lockutils [None req-96ee2dac-0a0f-4a64-9ce8-2aeff5240980 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77456) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 998.278887] env[77456]: DEBUG oslo_concurrency.lockutils [None req-96ee2dac-0a0f-4a64-9ce8-2aeff5240980 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77456) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 998.279243] env[77456]: DEBUG oslo_concurrency.lockutils [None req-96ee2dac-0a0f-4a64-9ce8-2aeff5240980 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77456) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 998.288480] env[77456]: DEBUG oslo_concurrency.lockutils [None req-96ee2dac-0a0f-4a64-9ce8-2aeff5240980 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77456) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 998.288737] env[77456]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-96ee2dac-0a0f-4a64-9ce8-2aeff5240980 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 998.288888] env[77456]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-96ee2dac-0a0f-4a64-9ce8-2aeff5240980 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77456) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1000.294986] env[77456]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-96ee2dac-0a0f-4a64-9ce8-2aeff5240980 None None] Session token - acquiring [ 1000.295359] env[77456]: DEBUG networking_nsxv3.common.locking [None req-96ee2dac-0a0f-4a64-9ce8-2aeff5240980 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77456) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1000.295565] env[77456]: DEBUG oslo_concurrency.lockutils [None req-96ee2dac-0a0f-4a64-9ce8-2aeff5240980 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77456) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1000.295691] env[77456]: DEBUG oslo_concurrency.lockutils [None req-96ee2dac-0a0f-4a64-9ce8-2aeff5240980 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77456) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1000.295999] env[77456]: DEBUG oslo_concurrency.lockutils [None req-96ee2dac-0a0f-4a64-9ce8-2aeff5240980 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77456) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1000.304887] env[77456]: DEBUG oslo_concurrency.lockutils [None req-96ee2dac-0a0f-4a64-9ce8-2aeff5240980 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77456) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1000.305196] env[77456]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-96ee2dac-0a0f-4a64-9ce8-2aeff5240980 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1000.305342] env[77456]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-96ee2dac-0a0f-4a64-9ce8-2aeff5240980 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77456) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1002.308251] env[77456]: CRITICAL neutron [None req-96ee2dac-0a0f-4a64-9ce8-2aeff5240980 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1002.308251] env[77456]: ERROR neutron Traceback (most recent call last): [ 1002.308251] env[77456]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 1002.308251] env[77456]: ERROR neutron sys.exit(main()) [ 1002.308251] env[77456]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 1002.308251] env[77456]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 1002.308251] env[77456]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 1002.308251] env[77456]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 1002.308251] env[77456]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 1002.308251] env[77456]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 1002.308251] env[77456]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 1002.308251] env[77456]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 1002.308251] env[77456]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 1002.308251] env[77456]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 1002.308251] env[77456]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 1002.308251] env[77456]: ERROR neutron response = self.get(path=path, params=params) [ 1002.309088] env[77456]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 1002.309088] env[77456]: ERROR neutron raise RuntimeError(msg, last_err) [ 1002.309088] env[77456]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1002.309088] env[77456]: ERROR neutron [ 1004.937372] env[77530]: INFO neutron.common.config [-] Logging enabled! [ 1004.937890] env[77530]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 1004.938370] env[77530]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77530) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 1004.938719] env[77530]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 1004.938911] env[77530]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 1004.939827] env[77530]: WARNING oslo_policy.policy [None req-112be503-3a7c-43c9-b4af-4941aabedff1 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1004.940178] env[77530]: WARNING oslo_policy.policy [None req-112be503-3a7c-43c9-b4af-4941aabedff1 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1004.941478] env[77530]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-112be503-3a7c-43c9-b4af-4941aabedff1 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 1004.955242] env[77530]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-112be503-3a7c-43c9-b4af-4941aabedff1 None None] Session token - acquiring [ 1004.956206] env[77530]: DEBUG networking_nsxv3.common.locking [None req-112be503-3a7c-43c9-b4af-4941aabedff1 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77530) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1004.956569] env[77530]: DEBUG oslo_concurrency.lockutils [None req-112be503-3a7c-43c9-b4af-4941aabedff1 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77530) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1004.956803] env[77530]: DEBUG oslo_concurrency.lockutils [None req-112be503-3a7c-43c9-b4af-4941aabedff1 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77530) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1004.957243] env[77530]: DEBUG oslo_concurrency.lockutils [None req-112be503-3a7c-43c9-b4af-4941aabedff1 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77530) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1004.966141] env[77530]: DEBUG oslo_concurrency.lockutils [None req-112be503-3a7c-43c9-b4af-4941aabedff1 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77530) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1004.966413] env[77530]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-112be503-3a7c-43c9-b4af-4941aabedff1 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1004.966568] env[77530]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-112be503-3a7c-43c9-b4af-4941aabedff1 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77530) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1006.974806] env[77530]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-112be503-3a7c-43c9-b4af-4941aabedff1 None None] Session token - acquiring [ 1006.975191] env[77530]: DEBUG networking_nsxv3.common.locking [None req-112be503-3a7c-43c9-b4af-4941aabedff1 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77530) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1006.975399] env[77530]: DEBUG oslo_concurrency.lockutils [None req-112be503-3a7c-43c9-b4af-4941aabedff1 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77530) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1006.975531] env[77530]: DEBUG oslo_concurrency.lockutils [None req-112be503-3a7c-43c9-b4af-4941aabedff1 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77530) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1006.975809] env[77530]: DEBUG oslo_concurrency.lockutils [None req-112be503-3a7c-43c9-b4af-4941aabedff1 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77530) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1006.985000] env[77530]: DEBUG oslo_concurrency.lockutils [None req-112be503-3a7c-43c9-b4af-4941aabedff1 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77530) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1006.985253] env[77530]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-112be503-3a7c-43c9-b4af-4941aabedff1 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1006.985393] env[77530]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-112be503-3a7c-43c9-b4af-4941aabedff1 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77530) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1008.988364] env[77530]: CRITICAL neutron [None req-112be503-3a7c-43c9-b4af-4941aabedff1 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1008.988364] env[77530]: ERROR neutron Traceback (most recent call last): [ 1008.988364] env[77530]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 1008.988364] env[77530]: ERROR neutron sys.exit(main()) [ 1008.988364] env[77530]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 1008.988364] env[77530]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 1008.988364] env[77530]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 1008.988364] env[77530]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 1008.988364] env[77530]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 1008.988364] env[77530]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 1008.988364] env[77530]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 1008.988364] env[77530]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 1008.988364] env[77530]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 1008.988364] env[77530]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 1008.988364] env[77530]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 1008.988364] env[77530]: ERROR neutron response = self.get(path=path, params=params) [ 1008.989065] env[77530]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 1008.989065] env[77530]: ERROR neutron raise RuntimeError(msg, last_err) [ 1008.989065] env[77530]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1008.989065] env[77530]: ERROR neutron [ 1011.539628] env[77597]: INFO neutron.common.config [-] Logging enabled! [ 1011.539995] env[77597]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 1011.540115] env[77597]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77597) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 1011.540335] env[77597]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 1011.540434] env[77597]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 1011.541242] env[77597]: WARNING oslo_policy.policy [None req-3a46d426-6c09-4127-9f23-12d10990ce60 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1011.541460] env[77597]: WARNING oslo_policy.policy [None req-3a46d426-6c09-4127-9f23-12d10990ce60 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1011.542715] env[77597]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-3a46d426-6c09-4127-9f23-12d10990ce60 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 1011.556399] env[77597]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3a46d426-6c09-4127-9f23-12d10990ce60 None None] Session token - acquiring [ 1011.557193] env[77597]: DEBUG networking_nsxv3.common.locking [None req-3a46d426-6c09-4127-9f23-12d10990ce60 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77597) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1011.557385] env[77597]: DEBUG oslo_concurrency.lockutils [None req-3a46d426-6c09-4127-9f23-12d10990ce60 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77597) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1011.557509] env[77597]: DEBUG oslo_concurrency.lockutils [None req-3a46d426-6c09-4127-9f23-12d10990ce60 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77597) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1011.557903] env[77597]: DEBUG oslo_concurrency.lockutils [None req-3a46d426-6c09-4127-9f23-12d10990ce60 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77597) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1011.566756] env[77597]: DEBUG oslo_concurrency.lockutils [None req-3a46d426-6c09-4127-9f23-12d10990ce60 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77597) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1011.567009] env[77597]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3a46d426-6c09-4127-9f23-12d10990ce60 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1011.567209] env[77597]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3a46d426-6c09-4127-9f23-12d10990ce60 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77597) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1013.573151] env[77597]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3a46d426-6c09-4127-9f23-12d10990ce60 None None] Session token - acquiring [ 1013.573478] env[77597]: DEBUG networking_nsxv3.common.locking [None req-3a46d426-6c09-4127-9f23-12d10990ce60 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77597) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1013.573776] env[77597]: DEBUG oslo_concurrency.lockutils [None req-3a46d426-6c09-4127-9f23-12d10990ce60 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77597) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1013.573776] env[77597]: DEBUG oslo_concurrency.lockutils [None req-3a46d426-6c09-4127-9f23-12d10990ce60 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77597) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1013.574093] env[77597]: DEBUG oslo_concurrency.lockutils [None req-3a46d426-6c09-4127-9f23-12d10990ce60 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77597) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1013.583500] env[77597]: DEBUG oslo_concurrency.lockutils [None req-3a46d426-6c09-4127-9f23-12d10990ce60 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77597) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1013.583748] env[77597]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3a46d426-6c09-4127-9f23-12d10990ce60 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1013.583914] env[77597]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3a46d426-6c09-4127-9f23-12d10990ce60 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77597) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1015.584940] env[77597]: CRITICAL neutron [None req-3a46d426-6c09-4127-9f23-12d10990ce60 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1015.584940] env[77597]: ERROR neutron Traceback (most recent call last): [ 1015.584940] env[77597]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 1015.584940] env[77597]: ERROR neutron sys.exit(main()) [ 1015.584940] env[77597]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 1015.584940] env[77597]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 1015.584940] env[77597]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 1015.584940] env[77597]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 1015.584940] env[77597]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 1015.584940] env[77597]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 1015.584940] env[77597]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 1015.584940] env[77597]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 1015.584940] env[77597]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 1015.584940] env[77597]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 1015.584940] env[77597]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 1015.584940] env[77597]: ERROR neutron response = self.get(path=path, params=params) [ 1015.585817] env[77597]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 1015.585817] env[77597]: ERROR neutron raise RuntimeError(msg, last_err) [ 1015.585817] env[77597]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1015.585817] env[77597]: ERROR neutron [ 1018.199635] env[77710]: INFO neutron.common.config [-] Logging enabled! [ 1018.199635] env[77710]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 1018.199635] env[77710]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77710) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 1018.199635] env[77710]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 1018.199635] env[77710]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 1018.199635] env[77710]: WARNING oslo_policy.policy [None req-8dfac8cb-670a-48d5-a0cb-246fc745cdc3 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1018.199635] env[77710]: WARNING oslo_policy.policy [None req-8dfac8cb-670a-48d5-a0cb-246fc745cdc3 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1018.199635] env[77710]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-8dfac8cb-670a-48d5-a0cb-246fc745cdc3 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 1018.209502] env[77710]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8dfac8cb-670a-48d5-a0cb-246fc745cdc3 None None] Session token - acquiring [ 1018.210362] env[77710]: DEBUG networking_nsxv3.common.locking [None req-8dfac8cb-670a-48d5-a0cb-246fc745cdc3 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77710) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1018.210569] env[77710]: DEBUG oslo_concurrency.lockutils [None req-8dfac8cb-670a-48d5-a0cb-246fc745cdc3 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77710) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1018.210695] env[77710]: DEBUG oslo_concurrency.lockutils [None req-8dfac8cb-670a-48d5-a0cb-246fc745cdc3 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77710) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1018.211064] env[77710]: DEBUG oslo_concurrency.lockutils [None req-8dfac8cb-670a-48d5-a0cb-246fc745cdc3 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77710) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1018.225134] env[77710]: DEBUG oslo_concurrency.lockutils [None req-8dfac8cb-670a-48d5-a0cb-246fc745cdc3 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77710) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1018.225134] env[77710]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8dfac8cb-670a-48d5-a0cb-246fc745cdc3 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1018.225134] env[77710]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8dfac8cb-670a-48d5-a0cb-246fc745cdc3 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77710) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1020.226752] env[77710]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8dfac8cb-670a-48d5-a0cb-246fc745cdc3 None None] Session token - acquiring [ 1020.227205] env[77710]: DEBUG networking_nsxv3.common.locking [None req-8dfac8cb-670a-48d5-a0cb-246fc745cdc3 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77710) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1020.227481] env[77710]: DEBUG oslo_concurrency.lockutils [None req-8dfac8cb-670a-48d5-a0cb-246fc745cdc3 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77710) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1020.227641] env[77710]: DEBUG oslo_concurrency.lockutils [None req-8dfac8cb-670a-48d5-a0cb-246fc745cdc3 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77710) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1020.227954] env[77710]: DEBUG oslo_concurrency.lockutils [None req-8dfac8cb-670a-48d5-a0cb-246fc745cdc3 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77710) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1020.237491] env[77710]: DEBUG oslo_concurrency.lockutils [None req-8dfac8cb-670a-48d5-a0cb-246fc745cdc3 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77710) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1020.237741] env[77710]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8dfac8cb-670a-48d5-a0cb-246fc745cdc3 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1020.237895] env[77710]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8dfac8cb-670a-48d5-a0cb-246fc745cdc3 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77710) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1022.240831] env[77710]: CRITICAL neutron [None req-8dfac8cb-670a-48d5-a0cb-246fc745cdc3 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1022.240831] env[77710]: ERROR neutron Traceback (most recent call last): [ 1022.240831] env[77710]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 1022.240831] env[77710]: ERROR neutron sys.exit(main()) [ 1022.240831] env[77710]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 1022.240831] env[77710]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 1022.240831] env[77710]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 1022.240831] env[77710]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 1022.240831] env[77710]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 1022.240831] env[77710]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 1022.240831] env[77710]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 1022.240831] env[77710]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 1022.240831] env[77710]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 1022.240831] env[77710]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 1022.240831] env[77710]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 1022.240831] env[77710]: ERROR neutron response = self.get(path=path, params=params) [ 1022.242369] env[77710]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 1022.242369] env[77710]: ERROR neutron raise RuntimeError(msg, last_err) [ 1022.242369] env[77710]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1022.242369] env[77710]: ERROR neutron [ 1024.688946] env[77910]: INFO neutron.common.config [-] Logging enabled! [ 1024.689315] env[77910]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 1024.689454] env[77910]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77910) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 1024.689677] env[77910]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 1024.689771] env[77910]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 1024.690599] env[77910]: WARNING oslo_policy.policy [None req-51bc4a47-37a4-4f13-8e19-14f95806bafa None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1024.690825] env[77910]: WARNING oslo_policy.policy [None req-51bc4a47-37a4-4f13-8e19-14f95806bafa None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1024.692025] env[77910]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-51bc4a47-37a4-4f13-8e19-14f95806bafa None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 1024.705948] env[77910]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-51bc4a47-37a4-4f13-8e19-14f95806bafa None None] Session token - acquiring [ 1024.706695] env[77910]: DEBUG networking_nsxv3.common.locking [None req-51bc4a47-37a4-4f13-8e19-14f95806bafa None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77910) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1024.706894] env[77910]: DEBUG oslo_concurrency.lockutils [None req-51bc4a47-37a4-4f13-8e19-14f95806bafa None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77910) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1024.707023] env[77910]: DEBUG oslo_concurrency.lockutils [None req-51bc4a47-37a4-4f13-8e19-14f95806bafa None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77910) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1024.707380] env[77910]: DEBUG oslo_concurrency.lockutils [None req-51bc4a47-37a4-4f13-8e19-14f95806bafa None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77910) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1024.716852] env[77910]: DEBUG oslo_concurrency.lockutils [None req-51bc4a47-37a4-4f13-8e19-14f95806bafa None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77910) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1024.717091] env[77910]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-51bc4a47-37a4-4f13-8e19-14f95806bafa None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1024.717250] env[77910]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-51bc4a47-37a4-4f13-8e19-14f95806bafa None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77910) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1026.723457] env[77910]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-51bc4a47-37a4-4f13-8e19-14f95806bafa None None] Session token - acquiring [ 1026.723905] env[77910]: DEBUG networking_nsxv3.common.locking [None req-51bc4a47-37a4-4f13-8e19-14f95806bafa None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77910) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1026.724046] env[77910]: DEBUG oslo_concurrency.lockutils [None req-51bc4a47-37a4-4f13-8e19-14f95806bafa None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77910) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1026.724117] env[77910]: DEBUG oslo_concurrency.lockutils [None req-51bc4a47-37a4-4f13-8e19-14f95806bafa None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77910) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1026.724435] env[77910]: DEBUG oslo_concurrency.lockutils [None req-51bc4a47-37a4-4f13-8e19-14f95806bafa None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77910) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1026.733656] env[77910]: DEBUG oslo_concurrency.lockutils [None req-51bc4a47-37a4-4f13-8e19-14f95806bafa None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77910) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1026.733926] env[77910]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-51bc4a47-37a4-4f13-8e19-14f95806bafa None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1026.734069] env[77910]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-51bc4a47-37a4-4f13-8e19-14f95806bafa None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77910) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1028.737311] env[77910]: CRITICAL neutron [None req-51bc4a47-37a4-4f13-8e19-14f95806bafa None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1028.737311] env[77910]: ERROR neutron Traceback (most recent call last): [ 1028.737311] env[77910]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 1028.737311] env[77910]: ERROR neutron sys.exit(main()) [ 1028.737311] env[77910]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 1028.737311] env[77910]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 1028.737311] env[77910]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 1028.737311] env[77910]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 1028.737311] env[77910]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 1028.737311] env[77910]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 1028.737311] env[77910]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 1028.737311] env[77910]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 1028.737311] env[77910]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 1028.737311] env[77910]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 1028.737311] env[77910]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 1028.737311] env[77910]: ERROR neutron response = self.get(path=path, params=params) [ 1028.738447] env[77910]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 1028.738447] env[77910]: ERROR neutron raise RuntimeError(msg, last_err) [ 1028.738447] env[77910]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1028.738447] env[77910]: ERROR neutron [ 1031.159525] env[78010]: INFO neutron.common.config [-] Logging enabled! [ 1031.160042] env[78010]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 1031.160042] env[78010]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=78010) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 1031.160255] env[78010]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 1031.160349] env[78010]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 1031.161176] env[78010]: WARNING oslo_policy.policy [None req-7ba4dec6-c7e9-49a4-a508-c260f7b6343b None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1031.161392] env[78010]: WARNING oslo_policy.policy [None req-7ba4dec6-c7e9-49a4-a508-c260f7b6343b None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1031.162617] env[78010]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-7ba4dec6-c7e9-49a4-a508-c260f7b6343b None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 1031.176651] env[78010]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7ba4dec6-c7e9-49a4-a508-c260f7b6343b None None] Session token - acquiring [ 1031.177404] env[78010]: DEBUG networking_nsxv3.common.locking [None req-7ba4dec6-c7e9-49a4-a508-c260f7b6343b None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=78010) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1031.177600] env[78010]: DEBUG oslo_concurrency.lockutils [None req-7ba4dec6-c7e9-49a4-a508-c260f7b6343b None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78010) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1031.177728] env[78010]: DEBUG oslo_concurrency.lockutils [None req-7ba4dec6-c7e9-49a4-a508-c260f7b6343b None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78010) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1031.178078] env[78010]: DEBUG oslo_concurrency.lockutils [None req-7ba4dec6-c7e9-49a4-a508-c260f7b6343b None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78010) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1031.186803] env[78010]: DEBUG oslo_concurrency.lockutils [None req-7ba4dec6-c7e9-49a4-a508-c260f7b6343b None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78010) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1031.187041] env[78010]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7ba4dec6-c7e9-49a4-a508-c260f7b6343b None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1031.187193] env[78010]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7ba4dec6-c7e9-49a4-a508-c260f7b6343b None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=78010) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1033.193226] env[78010]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7ba4dec6-c7e9-49a4-a508-c260f7b6343b None None] Session token - acquiring [ 1033.193635] env[78010]: DEBUG networking_nsxv3.common.locking [None req-7ba4dec6-c7e9-49a4-a508-c260f7b6343b None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=78010) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1033.193906] env[78010]: DEBUG oslo_concurrency.lockutils [None req-7ba4dec6-c7e9-49a4-a508-c260f7b6343b None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78010) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1033.194052] env[78010]: DEBUG oslo_concurrency.lockutils [None req-7ba4dec6-c7e9-49a4-a508-c260f7b6343b None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78010) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1033.194326] env[78010]: DEBUG oslo_concurrency.lockutils [None req-7ba4dec6-c7e9-49a4-a508-c260f7b6343b None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78010) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1033.203468] env[78010]: DEBUG oslo_concurrency.lockutils [None req-7ba4dec6-c7e9-49a4-a508-c260f7b6343b None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78010) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1033.203703] env[78010]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7ba4dec6-c7e9-49a4-a508-c260f7b6343b None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1033.203920] env[78010]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7ba4dec6-c7e9-49a4-a508-c260f7b6343b None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=78010) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1035.204945] env[78010]: CRITICAL neutron [None req-7ba4dec6-c7e9-49a4-a508-c260f7b6343b None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1035.204945] env[78010]: ERROR neutron Traceback (most recent call last): [ 1035.204945] env[78010]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 1035.204945] env[78010]: ERROR neutron sys.exit(main()) [ 1035.204945] env[78010]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 1035.204945] env[78010]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 1035.204945] env[78010]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 1035.204945] env[78010]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 1035.204945] env[78010]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 1035.204945] env[78010]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 1035.204945] env[78010]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 1035.204945] env[78010]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 1035.204945] env[78010]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 1035.204945] env[78010]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 1035.204945] env[78010]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 1035.204945] env[78010]: ERROR neutron response = self.get(path=path, params=params) [ 1035.205768] env[78010]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 1035.205768] env[78010]: ERROR neutron raise RuntimeError(msg, last_err) [ 1035.205768] env[78010]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1035.205768] env[78010]: ERROR neutron [ 1037.635372] env[78100]: INFO neutron.common.config [-] Logging enabled! [ 1037.635775] env[78100]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 1037.635893] env[78100]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=78100) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 1037.636149] env[78100]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 1037.636247] env[78100]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 1037.637070] env[78100]: WARNING oslo_policy.policy [None req-1b1f161a-f620-4350-b0d8-15895e61e0b3 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1037.637320] env[78100]: WARNING oslo_policy.policy [None req-1b1f161a-f620-4350-b0d8-15895e61e0b3 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1037.638500] env[78100]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-1b1f161a-f620-4350-b0d8-15895e61e0b3 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 1037.653325] env[78100]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1b1f161a-f620-4350-b0d8-15895e61e0b3 None None] Session token - acquiring [ 1037.654093] env[78100]: DEBUG networking_nsxv3.common.locking [None req-1b1f161a-f620-4350-b0d8-15895e61e0b3 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=78100) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1037.654282] env[78100]: DEBUG oslo_concurrency.lockutils [None req-1b1f161a-f620-4350-b0d8-15895e61e0b3 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78100) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1037.654408] env[78100]: DEBUG oslo_concurrency.lockutils [None req-1b1f161a-f620-4350-b0d8-15895e61e0b3 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78100) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1037.654758] env[78100]: DEBUG oslo_concurrency.lockutils [None req-1b1f161a-f620-4350-b0d8-15895e61e0b3 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78100) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1037.663719] env[78100]: DEBUG oslo_concurrency.lockutils [None req-1b1f161a-f620-4350-b0d8-15895e61e0b3 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78100) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1037.663979] env[78100]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1b1f161a-f620-4350-b0d8-15895e61e0b3 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1037.664152] env[78100]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1b1f161a-f620-4350-b0d8-15895e61e0b3 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=78100) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1039.670208] env[78100]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1b1f161a-f620-4350-b0d8-15895e61e0b3 None None] Session token - acquiring [ 1039.670558] env[78100]: DEBUG networking_nsxv3.common.locking [None req-1b1f161a-f620-4350-b0d8-15895e61e0b3 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=78100) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1039.670715] env[78100]: DEBUG oslo_concurrency.lockutils [None req-1b1f161a-f620-4350-b0d8-15895e61e0b3 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78100) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1039.670797] env[78100]: DEBUG oslo_concurrency.lockutils [None req-1b1f161a-f620-4350-b0d8-15895e61e0b3 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78100) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1039.671081] env[78100]: DEBUG oslo_concurrency.lockutils [None req-1b1f161a-f620-4350-b0d8-15895e61e0b3 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78100) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1039.679835] env[78100]: DEBUG oslo_concurrency.lockutils [None req-1b1f161a-f620-4350-b0d8-15895e61e0b3 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78100) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1039.680080] env[78100]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1b1f161a-f620-4350-b0d8-15895e61e0b3 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1039.680223] env[78100]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1b1f161a-f620-4350-b0d8-15895e61e0b3 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=78100) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1041.683168] env[78100]: CRITICAL neutron [None req-1b1f161a-f620-4350-b0d8-15895e61e0b3 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1041.683168] env[78100]: ERROR neutron Traceback (most recent call last): [ 1041.683168] env[78100]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 1041.683168] env[78100]: ERROR neutron sys.exit(main()) [ 1041.683168] env[78100]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 1041.683168] env[78100]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 1041.683168] env[78100]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 1041.683168] env[78100]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 1041.683168] env[78100]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 1041.683168] env[78100]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 1041.683168] env[78100]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 1041.683168] env[78100]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 1041.683168] env[78100]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 1041.683168] env[78100]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 1041.683168] env[78100]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 1041.683168] env[78100]: ERROR neutron response = self.get(path=path, params=params) [ 1041.684317] env[78100]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 1041.684317] env[78100]: ERROR neutron raise RuntimeError(msg, last_err) [ 1041.684317] env[78100]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1041.684317] env[78100]: ERROR neutron [ 1044.143575] env[78109]: INFO neutron.common.config [-] Logging enabled! [ 1044.144035] env[78109]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 1044.144130] env[78109]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=78109) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 1044.144377] env[78109]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 1044.144474] env[78109]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 1044.145333] env[78109]: WARNING oslo_policy.policy [None req-46e10016-fbe5-4f8e-9b5c-093ca807b9e5 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1044.145560] env[78109]: WARNING oslo_policy.policy [None req-46e10016-fbe5-4f8e-9b5c-093ca807b9e5 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1044.146711] env[78109]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-46e10016-fbe5-4f8e-9b5c-093ca807b9e5 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 1044.160633] env[78109]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-46e10016-fbe5-4f8e-9b5c-093ca807b9e5 None None] Session token - acquiring [ 1044.161382] env[78109]: DEBUG networking_nsxv3.common.locking [None req-46e10016-fbe5-4f8e-9b5c-093ca807b9e5 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=78109) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1044.161579] env[78109]: DEBUG oslo_concurrency.lockutils [None req-46e10016-fbe5-4f8e-9b5c-093ca807b9e5 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78109) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1044.161705] env[78109]: DEBUG oslo_concurrency.lockutils [None req-46e10016-fbe5-4f8e-9b5c-093ca807b9e5 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78109) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1044.162061] env[78109]: DEBUG oslo_concurrency.lockutils [None req-46e10016-fbe5-4f8e-9b5c-093ca807b9e5 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78109) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1044.170694] env[78109]: DEBUG oslo_concurrency.lockutils [None req-46e10016-fbe5-4f8e-9b5c-093ca807b9e5 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78109) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1044.170932] env[78109]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-46e10016-fbe5-4f8e-9b5c-093ca807b9e5 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1044.171083] env[78109]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-46e10016-fbe5-4f8e-9b5c-093ca807b9e5 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=78109) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1046.177079] env[78109]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-46e10016-fbe5-4f8e-9b5c-093ca807b9e5 None None] Session token - acquiring [ 1046.177541] env[78109]: DEBUG networking_nsxv3.common.locking [None req-46e10016-fbe5-4f8e-9b5c-093ca807b9e5 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=78109) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1046.177541] env[78109]: DEBUG oslo_concurrency.lockutils [None req-46e10016-fbe5-4f8e-9b5c-093ca807b9e5 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78109) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1046.177704] env[78109]: DEBUG oslo_concurrency.lockutils [None req-46e10016-fbe5-4f8e-9b5c-093ca807b9e5 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78109) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1046.177972] env[78109]: DEBUG oslo_concurrency.lockutils [None req-46e10016-fbe5-4f8e-9b5c-093ca807b9e5 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78109) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1046.186819] env[78109]: DEBUG oslo_concurrency.lockutils [None req-46e10016-fbe5-4f8e-9b5c-093ca807b9e5 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78109) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1046.187044] env[78109]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-46e10016-fbe5-4f8e-9b5c-093ca807b9e5 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1046.187186] env[78109]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-46e10016-fbe5-4f8e-9b5c-093ca807b9e5 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=78109) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1048.190153] env[78109]: CRITICAL neutron [None req-46e10016-fbe5-4f8e-9b5c-093ca807b9e5 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1048.190153] env[78109]: ERROR neutron Traceback (most recent call last): [ 1048.190153] env[78109]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 1048.190153] env[78109]: ERROR neutron sys.exit(main()) [ 1048.190153] env[78109]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 1048.190153] env[78109]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 1048.190153] env[78109]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 1048.190153] env[78109]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 1048.190153] env[78109]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 1048.190153] env[78109]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 1048.190153] env[78109]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 1048.190153] env[78109]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 1048.190153] env[78109]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 1048.190153] env[78109]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 1048.190153] env[78109]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 1048.190153] env[78109]: ERROR neutron response = self.get(path=path, params=params) [ 1048.191029] env[78109]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 1048.191029] env[78109]: ERROR neutron raise RuntimeError(msg, last_err) [ 1048.191029] env[78109]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1048.191029] env[78109]: ERROR neutron [ 1050.653578] env[78115]: INFO neutron.common.config [-] Logging enabled! [ 1050.653956] env[78115]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev45 [ 1050.654111] env[78115]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=78115) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 1050.654337] env[78115]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 1050.654432] env[78115]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 1050.655269] env[78115]: WARNING oslo_policy.policy [None req-9c854994-cdb7-4e18-8e2b-b0fb0cf40a54 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1050.655489] env[78115]: WARNING oslo_policy.policy [None req-9c854994-cdb7-4e18-8e2b-b0fb0cf40a54 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1050.656701] env[78115]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-9c854994-cdb7-4e18-8e2b-b0fb0cf40a54 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 1050.670819] env[78115]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9c854994-cdb7-4e18-8e2b-b0fb0cf40a54 None None] Session token - acquiring [ 1050.671587] env[78115]: DEBUG networking_nsxv3.common.locking [None req-9c854994-cdb7-4e18-8e2b-b0fb0cf40a54 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=78115) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1050.671789] env[78115]: DEBUG oslo_concurrency.lockutils [None req-9c854994-cdb7-4e18-8e2b-b0fb0cf40a54 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78115) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1050.671919] env[78115]: DEBUG oslo_concurrency.lockutils [None req-9c854994-cdb7-4e18-8e2b-b0fb0cf40a54 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78115) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1050.672288] env[78115]: DEBUG oslo_concurrency.lockutils [None req-9c854994-cdb7-4e18-8e2b-b0fb0cf40a54 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78115) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1050.681174] env[78115]: DEBUG oslo_concurrency.lockutils [None req-9c854994-cdb7-4e18-8e2b-b0fb0cf40a54 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78115) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1050.681424] env[78115]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9c854994-cdb7-4e18-8e2b-b0fb0cf40a54 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1050.681575] env[78115]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9c854994-cdb7-4e18-8e2b-b0fb0cf40a54 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=78115) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1052.687533] env[78115]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9c854994-cdb7-4e18-8e2b-b0fb0cf40a54 None None] Session token - acquiring [ 1052.687913] env[78115]: DEBUG networking_nsxv3.common.locking [None req-9c854994-cdb7-4e18-8e2b-b0fb0cf40a54 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=78115) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1052.688091] env[78115]: DEBUG oslo_concurrency.lockutils [None req-9c854994-cdb7-4e18-8e2b-b0fb0cf40a54 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78115) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1052.688162] env[78115]: DEBUG oslo_concurrency.lockutils [None req-9c854994-cdb7-4e18-8e2b-b0fb0cf40a54 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78115) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1052.688437] env[78115]: DEBUG oslo_concurrency.lockutils [None req-9c854994-cdb7-4e18-8e2b-b0fb0cf40a54 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78115) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1052.697234] env[78115]: DEBUG oslo_concurrency.lockutils [None req-9c854994-cdb7-4e18-8e2b-b0fb0cf40a54 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=78115) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1052.697458] env[78115]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9c854994-cdb7-4e18-8e2b-b0fb0cf40a54 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1052.697596] env[78115]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9c854994-cdb7-4e18-8e2b-b0fb0cf40a54 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=78115) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}}