[ 452.009869] env[61421]: INFO neutron.common.config [-] Logging enabled! [ 452.010361] env[61421]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 452.010361] env[61421]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=61421) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 452.010566] env[61421]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 452.010657] env[61421]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 452.011457] env[61421]: WARNING oslo_policy.policy [None req-400ee9c3-b0dd-4001-b460-e62c60f50052 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 452.011674] env[61421]: WARNING oslo_policy.policy [None req-400ee9c3-b0dd-4001-b460-e62c60f50052 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 452.012892] env[61421]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-400ee9c3-b0dd-4001-b460-e62c60f50052 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 452.035303] env[61421]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-400ee9c3-b0dd-4001-b460-e62c60f50052 None None] Session token - acquiring [ 452.036090] env[61421]: DEBUG networking_nsxv3.common.locking [None req-400ee9c3-b0dd-4001-b460-e62c60f50052 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=61421) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 452.036297] env[61421]: DEBUG oslo_concurrency.lockutils [None req-400ee9c3-b0dd-4001-b460-e62c60f50052 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61421) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 452.036422] env[61421]: DEBUG oslo_concurrency.lockutils [None req-400ee9c3-b0dd-4001-b460-e62c60f50052 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61421) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 452.036828] env[61421]: DEBUG oslo_concurrency.lockutils [None req-400ee9c3-b0dd-4001-b460-e62c60f50052 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61421) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 452.046217] env[61421]: DEBUG oslo_concurrency.lockutils [None req-400ee9c3-b0dd-4001-b460-e62c60f50052 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61421) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 452.046451] env[61421]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-400ee9c3-b0dd-4001-b460-e62c60f50052 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 452.046595] env[61421]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-400ee9c3-b0dd-4001-b460-e62c60f50052 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=61421) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 454.052750] env[61421]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-400ee9c3-b0dd-4001-b460-e62c60f50052 None None] Session token - acquiring [ 454.053164] env[61421]: DEBUG networking_nsxv3.common.locking [None req-400ee9c3-b0dd-4001-b460-e62c60f50052 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=61421) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 454.053401] env[61421]: DEBUG oslo_concurrency.lockutils [None req-400ee9c3-b0dd-4001-b460-e62c60f50052 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61421) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 454.053482] env[61421]: DEBUG oslo_concurrency.lockutils [None req-400ee9c3-b0dd-4001-b460-e62c60f50052 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61421) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 454.053842] env[61421]: DEBUG oslo_concurrency.lockutils [None req-400ee9c3-b0dd-4001-b460-e62c60f50052 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61421) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 454.064358] env[61421]: DEBUG oslo_concurrency.lockutils [None req-400ee9c3-b0dd-4001-b460-e62c60f50052 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61421) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 454.064599] env[61421]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-400ee9c3-b0dd-4001-b460-e62c60f50052 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 454.064743] env[61421]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-400ee9c3-b0dd-4001-b460-e62c60f50052 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=61421) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 456.067740] env[61421]: CRITICAL neutron [None req-400ee9c3-b0dd-4001-b460-e62c60f50052 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 456.067740] env[61421]: ERROR neutron Traceback (most recent call last): [ 456.067740] env[61421]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 456.067740] env[61421]: ERROR neutron sys.exit(main()) [ 456.067740] env[61421]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 456.067740] env[61421]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 456.067740] env[61421]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 456.067740] env[61421]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 456.067740] env[61421]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 456.067740] env[61421]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 456.067740] env[61421]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 456.067740] env[61421]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 456.067740] env[61421]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 456.067740] env[61421]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 456.067740] env[61421]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 456.067740] env[61421]: ERROR neutron response = self.get(path=path, params=params) [ 456.068784] env[61421]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 456.068784] env[61421]: ERROR neutron raise RuntimeError(msg, last_err) [ 456.068784] env[61421]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 456.068784] env[61421]: ERROR neutron [ 458.599269] env[61454]: INFO neutron.common.config [-] Logging enabled! [ 458.599658] env[61454]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 458.599762] env[61454]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=61454) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 458.599983] env[61454]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 458.600129] env[61454]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 458.600964] env[61454]: WARNING oslo_policy.policy [None req-b1543bdf-8162-455b-b63d-9105874a15a5 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 458.601179] env[61454]: WARNING oslo_policy.policy [None req-b1543bdf-8162-455b-b63d-9105874a15a5 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 458.602364] env[61454]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-b1543bdf-8162-455b-b63d-9105874a15a5 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 458.617435] env[61454]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b1543bdf-8162-455b-b63d-9105874a15a5 None None] Session token - acquiring [ 458.618151] env[61454]: DEBUG networking_nsxv3.common.locking [None req-b1543bdf-8162-455b-b63d-9105874a15a5 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=61454) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 458.618345] env[61454]: DEBUG oslo_concurrency.lockutils [None req-b1543bdf-8162-455b-b63d-9105874a15a5 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61454) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 458.618472] env[61454]: DEBUG oslo_concurrency.lockutils [None req-b1543bdf-8162-455b-b63d-9105874a15a5 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61454) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 458.618822] env[61454]: DEBUG oslo_concurrency.lockutils [None req-b1543bdf-8162-455b-b63d-9105874a15a5 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61454) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 458.628544] env[61454]: DEBUG oslo_concurrency.lockutils [None req-b1543bdf-8162-455b-b63d-9105874a15a5 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61454) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 458.628747] env[61454]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b1543bdf-8162-455b-b63d-9105874a15a5 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 458.628893] env[61454]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b1543bdf-8162-455b-b63d-9105874a15a5 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=61454) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 460.634881] env[61454]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b1543bdf-8162-455b-b63d-9105874a15a5 None None] Session token - acquiring [ 460.635133] env[61454]: DEBUG networking_nsxv3.common.locking [None req-b1543bdf-8162-455b-b63d-9105874a15a5 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=61454) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 460.635320] env[61454]: DEBUG oslo_concurrency.lockutils [None req-b1543bdf-8162-455b-b63d-9105874a15a5 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61454) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 460.635442] env[61454]: DEBUG oslo_concurrency.lockutils [None req-b1543bdf-8162-455b-b63d-9105874a15a5 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61454) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 460.635723] env[61454]: DEBUG oslo_concurrency.lockutils [None req-b1543bdf-8162-455b-b63d-9105874a15a5 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61454) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 460.645456] env[61454]: DEBUG oslo_concurrency.lockutils [None req-b1543bdf-8162-455b-b63d-9105874a15a5 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61454) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 460.645693] env[61454]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b1543bdf-8162-455b-b63d-9105874a15a5 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 460.645834] env[61454]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b1543bdf-8162-455b-b63d-9105874a15a5 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=61454) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 462.648763] env[61454]: CRITICAL neutron [None req-b1543bdf-8162-455b-b63d-9105874a15a5 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 462.648763] env[61454]: ERROR neutron Traceback (most recent call last): [ 462.648763] env[61454]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 462.648763] env[61454]: ERROR neutron sys.exit(main()) [ 462.648763] env[61454]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 462.648763] env[61454]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 462.648763] env[61454]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 462.648763] env[61454]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 462.648763] env[61454]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 462.648763] env[61454]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 462.648763] env[61454]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 462.648763] env[61454]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 462.648763] env[61454]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 462.648763] env[61454]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 462.648763] env[61454]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 462.648763] env[61454]: ERROR neutron response = self.get(path=path, params=params) [ 462.649349] env[61454]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 462.649349] env[61454]: ERROR neutron raise RuntimeError(msg, last_err) [ 462.649349] env[61454]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 462.649349] env[61454]: ERROR neutron [ 465.050296] env[61479]: INFO neutron.common.config [-] Logging enabled! [ 465.050685] env[61479]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 465.050793] env[61479]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=61479) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 465.050984] env[61479]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 465.051089] env[61479]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 465.051952] env[61479]: WARNING oslo_policy.policy [None req-e6df1a77-2883-43c1-815c-6e586fb57c93 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 465.052227] env[61479]: WARNING oslo_policy.policy [None req-e6df1a77-2883-43c1-815c-6e586fb57c93 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 465.053396] env[61479]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-e6df1a77-2883-43c1-815c-6e586fb57c93 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 465.067350] env[61479]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e6df1a77-2883-43c1-815c-6e586fb57c93 None None] Session token - acquiring [ 465.068108] env[61479]: DEBUG networking_nsxv3.common.locking [None req-e6df1a77-2883-43c1-815c-6e586fb57c93 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=61479) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 465.068308] env[61479]: DEBUG oslo_concurrency.lockutils [None req-e6df1a77-2883-43c1-815c-6e586fb57c93 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61479) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 465.068436] env[61479]: DEBUG oslo_concurrency.lockutils [None req-e6df1a77-2883-43c1-815c-6e586fb57c93 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61479) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 465.068789] env[61479]: DEBUG oslo_concurrency.lockutils [None req-e6df1a77-2883-43c1-815c-6e586fb57c93 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61479) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 465.078288] env[61479]: DEBUG oslo_concurrency.lockutils [None req-e6df1a77-2883-43c1-815c-6e586fb57c93 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61479) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 465.078531] env[61479]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e6df1a77-2883-43c1-815c-6e586fb57c93 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 465.078677] env[61479]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e6df1a77-2883-43c1-815c-6e586fb57c93 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=61479) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 467.084705] env[61479]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e6df1a77-2883-43c1-815c-6e586fb57c93 None None] Session token - acquiring [ 467.085012] env[61479]: DEBUG networking_nsxv3.common.locking [None req-e6df1a77-2883-43c1-815c-6e586fb57c93 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=61479) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 467.085202] env[61479]: DEBUG oslo_concurrency.lockutils [None req-e6df1a77-2883-43c1-815c-6e586fb57c93 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61479) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 467.085327] env[61479]: DEBUG oslo_concurrency.lockutils [None req-e6df1a77-2883-43c1-815c-6e586fb57c93 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61479) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 467.085611] env[61479]: DEBUG oslo_concurrency.lockutils [None req-e6df1a77-2883-43c1-815c-6e586fb57c93 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61479) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 467.094953] env[61479]: DEBUG oslo_concurrency.lockutils [None req-e6df1a77-2883-43c1-815c-6e586fb57c93 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61479) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 467.095294] env[61479]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e6df1a77-2883-43c1-815c-6e586fb57c93 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 467.095453] env[61479]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e6df1a77-2883-43c1-815c-6e586fb57c93 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=61479) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 469.098399] env[61479]: CRITICAL neutron [None req-e6df1a77-2883-43c1-815c-6e586fb57c93 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 469.098399] env[61479]: ERROR neutron Traceback (most recent call last): [ 469.098399] env[61479]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 469.098399] env[61479]: ERROR neutron sys.exit(main()) [ 469.098399] env[61479]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 469.098399] env[61479]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 469.098399] env[61479]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 469.098399] env[61479]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 469.098399] env[61479]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 469.098399] env[61479]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 469.098399] env[61479]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 469.098399] env[61479]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 469.098399] env[61479]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 469.098399] env[61479]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 469.098399] env[61479]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 469.098399] env[61479]: ERROR neutron response = self.get(path=path, params=params) [ 469.099208] env[61479]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 469.099208] env[61479]: ERROR neutron raise RuntimeError(msg, last_err) [ 469.099208] env[61479]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 469.099208] env[61479]: ERROR neutron [ 471.482562] env[61524]: INFO neutron.common.config [-] Logging enabled! [ 471.483017] env[61524]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 471.483064] env[61524]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=61524) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 471.483298] env[61524]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 471.483395] env[61524]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 471.484393] env[61524]: WARNING oslo_policy.policy [None req-0b56ffe7-29f5-4c6b-8c3d-008d6034371a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 471.484695] env[61524]: WARNING oslo_policy.policy [None req-0b56ffe7-29f5-4c6b-8c3d-008d6034371a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 471.485956] env[61524]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-0b56ffe7-29f5-4c6b-8c3d-008d6034371a None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 471.500192] env[61524]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0b56ffe7-29f5-4c6b-8c3d-008d6034371a None None] Session token - acquiring [ 471.500937] env[61524]: DEBUG networking_nsxv3.common.locking [None req-0b56ffe7-29f5-4c6b-8c3d-008d6034371a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=61524) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 471.501139] env[61524]: DEBUG oslo_concurrency.lockutils [None req-0b56ffe7-29f5-4c6b-8c3d-008d6034371a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61524) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 471.501268] env[61524]: DEBUG oslo_concurrency.lockutils [None req-0b56ffe7-29f5-4c6b-8c3d-008d6034371a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61524) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 471.501641] env[61524]: DEBUG oslo_concurrency.lockutils [None req-0b56ffe7-29f5-4c6b-8c3d-008d6034371a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61524) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 471.510385] env[61524]: DEBUG oslo_concurrency.lockutils [None req-0b56ffe7-29f5-4c6b-8c3d-008d6034371a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61524) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 471.510632] env[61524]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0b56ffe7-29f5-4c6b-8c3d-008d6034371a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 471.510782] env[61524]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0b56ffe7-29f5-4c6b-8c3d-008d6034371a None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=61524) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 473.517055] env[61524]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0b56ffe7-29f5-4c6b-8c3d-008d6034371a None None] Session token - acquiring [ 473.517366] env[61524]: DEBUG networking_nsxv3.common.locking [None req-0b56ffe7-29f5-4c6b-8c3d-008d6034371a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=61524) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 473.517508] env[61524]: DEBUG oslo_concurrency.lockutils [None req-0b56ffe7-29f5-4c6b-8c3d-008d6034371a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61524) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 473.517640] env[61524]: DEBUG oslo_concurrency.lockutils [None req-0b56ffe7-29f5-4c6b-8c3d-008d6034371a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61524) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 473.517912] env[61524]: DEBUG oslo_concurrency.lockutils [None req-0b56ffe7-29f5-4c6b-8c3d-008d6034371a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61524) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 473.526820] env[61524]: DEBUG oslo_concurrency.lockutils [None req-0b56ffe7-29f5-4c6b-8c3d-008d6034371a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61524) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 473.527050] env[61524]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0b56ffe7-29f5-4c6b-8c3d-008d6034371a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 473.527189] env[61524]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0b56ffe7-29f5-4c6b-8c3d-008d6034371a None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=61524) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 475.530191] env[61524]: CRITICAL neutron [None req-0b56ffe7-29f5-4c6b-8c3d-008d6034371a None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 475.530191] env[61524]: ERROR neutron Traceback (most recent call last): [ 475.530191] env[61524]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 475.530191] env[61524]: ERROR neutron sys.exit(main()) [ 475.530191] env[61524]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 475.530191] env[61524]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 475.530191] env[61524]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 475.530191] env[61524]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 475.530191] env[61524]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 475.530191] env[61524]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 475.530191] env[61524]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 475.530191] env[61524]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 475.530191] env[61524]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 475.530191] env[61524]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 475.530191] env[61524]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 475.530191] env[61524]: ERROR neutron response = self.get(path=path, params=params) [ 475.531070] env[61524]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 475.531070] env[61524]: ERROR neutron raise RuntimeError(msg, last_err) [ 475.531070] env[61524]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 475.531070] env[61524]: ERROR neutron [ 477.975165] env[61531]: INFO neutron.common.config [-] Logging enabled! [ 477.975565] env[61531]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 477.975886] env[61531]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=61531) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 477.975886] env[61531]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 477.975999] env[61531]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 477.976808] env[61531]: WARNING oslo_policy.policy [None req-f724a3d8-6c43-4caa-b793-5e62e2e670c2 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 477.977915] env[61531]: WARNING oslo_policy.policy [None req-f724a3d8-6c43-4caa-b793-5e62e2e670c2 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 477.978172] env[61531]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-f724a3d8-6c43-4caa-b793-5e62e2e670c2 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 477.992443] env[61531]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f724a3d8-6c43-4caa-b793-5e62e2e670c2 None None] Session token - acquiring [ 477.993161] env[61531]: DEBUG networking_nsxv3.common.locking [None req-f724a3d8-6c43-4caa-b793-5e62e2e670c2 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=61531) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 477.993354] env[61531]: DEBUG oslo_concurrency.lockutils [None req-f724a3d8-6c43-4caa-b793-5e62e2e670c2 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61531) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 477.993480] env[61531]: DEBUG oslo_concurrency.lockutils [None req-f724a3d8-6c43-4caa-b793-5e62e2e670c2 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61531) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 477.993875] env[61531]: DEBUG oslo_concurrency.lockutils [None req-f724a3d8-6c43-4caa-b793-5e62e2e670c2 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61531) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 478.002700] env[61531]: DEBUG oslo_concurrency.lockutils [None req-f724a3d8-6c43-4caa-b793-5e62e2e670c2 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61531) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 478.002938] env[61531]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f724a3d8-6c43-4caa-b793-5e62e2e670c2 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 478.003080] env[61531]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f724a3d8-6c43-4caa-b793-5e62e2e670c2 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=61531) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 480.009186] env[61531]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f724a3d8-6c43-4caa-b793-5e62e2e670c2 None None] Session token - acquiring [ 480.009489] env[61531]: DEBUG networking_nsxv3.common.locking [None req-f724a3d8-6c43-4caa-b793-5e62e2e670c2 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=61531) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 480.009653] env[61531]: DEBUG oslo_concurrency.lockutils [None req-f724a3d8-6c43-4caa-b793-5e62e2e670c2 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61531) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 480.009775] env[61531]: DEBUG oslo_concurrency.lockutils [None req-f724a3d8-6c43-4caa-b793-5e62e2e670c2 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61531) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 480.010054] env[61531]: DEBUG oslo_concurrency.lockutils [None req-f724a3d8-6c43-4caa-b793-5e62e2e670c2 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61531) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 480.019251] env[61531]: DEBUG oslo_concurrency.lockutils [None req-f724a3d8-6c43-4caa-b793-5e62e2e670c2 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61531) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 480.019486] env[61531]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f724a3d8-6c43-4caa-b793-5e62e2e670c2 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 480.019625] env[61531]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f724a3d8-6c43-4caa-b793-5e62e2e670c2 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=61531) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 482.022733] env[61531]: CRITICAL neutron [None req-f724a3d8-6c43-4caa-b793-5e62e2e670c2 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 482.022733] env[61531]: ERROR neutron Traceback (most recent call last): [ 482.022733] env[61531]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 482.022733] env[61531]: ERROR neutron sys.exit(main()) [ 482.022733] env[61531]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 482.022733] env[61531]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 482.022733] env[61531]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 482.022733] env[61531]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 482.022733] env[61531]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 482.022733] env[61531]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 482.022733] env[61531]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 482.022733] env[61531]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 482.022733] env[61531]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 482.022733] env[61531]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 482.022733] env[61531]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 482.022733] env[61531]: ERROR neutron response = self.get(path=path, params=params) [ 482.023471] env[61531]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 482.023471] env[61531]: ERROR neutron raise RuntimeError(msg, last_err) [ 482.023471] env[61531]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 482.023471] env[61531]: ERROR neutron [ 484.521139] env[61539]: INFO neutron.common.config [-] Logging enabled! [ 484.521468] env[61539]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 484.521655] env[61539]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=61539) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 484.521882] env[61539]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 484.521976] env[61539]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 484.522797] env[61539]: WARNING oslo_policy.policy [None req-95b09721-d028-4720-a554-c1e73b95c557 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 484.523016] env[61539]: WARNING oslo_policy.policy [None req-95b09721-d028-4720-a554-c1e73b95c557 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 484.524232] env[61539]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-95b09721-d028-4720-a554-c1e73b95c557 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 484.538319] env[61539]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-95b09721-d028-4720-a554-c1e73b95c557 None None] Session token - acquiring [ 484.539059] env[61539]: DEBUG networking_nsxv3.common.locking [None req-95b09721-d028-4720-a554-c1e73b95c557 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=61539) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 484.539255] env[61539]: DEBUG oslo_concurrency.lockutils [None req-95b09721-d028-4720-a554-c1e73b95c557 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61539) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 484.539380] env[61539]: DEBUG oslo_concurrency.lockutils [None req-95b09721-d028-4720-a554-c1e73b95c557 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61539) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 484.539730] env[61539]: DEBUG oslo_concurrency.lockutils [None req-95b09721-d028-4720-a554-c1e73b95c557 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61539) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 484.548768] env[61539]: DEBUG oslo_concurrency.lockutils [None req-95b09721-d028-4720-a554-c1e73b95c557 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61539) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 484.549014] env[61539]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-95b09721-d028-4720-a554-c1e73b95c557 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 484.549166] env[61539]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-95b09721-d028-4720-a554-c1e73b95c557 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=61539) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 486.555784] env[61539]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-95b09721-d028-4720-a554-c1e73b95c557 None None] Session token - acquiring [ 486.556256] env[61539]: DEBUG networking_nsxv3.common.locking [None req-95b09721-d028-4720-a554-c1e73b95c557 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=61539) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 486.556321] env[61539]: DEBUG oslo_concurrency.lockutils [None req-95b09721-d028-4720-a554-c1e73b95c557 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61539) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 486.556481] env[61539]: DEBUG oslo_concurrency.lockutils [None req-95b09721-d028-4720-a554-c1e73b95c557 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61539) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 486.556699] env[61539]: DEBUG oslo_concurrency.lockutils [None req-95b09721-d028-4720-a554-c1e73b95c557 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61539) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 486.565840] env[61539]: DEBUG oslo_concurrency.lockutils [None req-95b09721-d028-4720-a554-c1e73b95c557 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61539) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 486.566121] env[61539]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-95b09721-d028-4720-a554-c1e73b95c557 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 486.566262] env[61539]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-95b09721-d028-4720-a554-c1e73b95c557 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=61539) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 488.568906] env[61539]: CRITICAL neutron [None req-95b09721-d028-4720-a554-c1e73b95c557 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 488.568906] env[61539]: ERROR neutron Traceback (most recent call last): [ 488.568906] env[61539]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 488.568906] env[61539]: ERROR neutron sys.exit(main()) [ 488.568906] env[61539]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 488.568906] env[61539]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 488.568906] env[61539]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 488.568906] env[61539]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 488.568906] env[61539]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 488.568906] env[61539]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 488.568906] env[61539]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 488.568906] env[61539]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 488.568906] env[61539]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 488.568906] env[61539]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 488.568906] env[61539]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 488.568906] env[61539]: ERROR neutron response = self.get(path=path, params=params) [ 488.569658] env[61539]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 488.569658] env[61539]: ERROR neutron raise RuntimeError(msg, last_err) [ 488.569658] env[61539]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 488.569658] env[61539]: ERROR neutron [ 491.213795] env[61770]: INFO neutron.common.config [-] Logging enabled! [ 491.214076] env[61770]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 491.214312] env[61770]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=61770) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 491.214537] env[61770]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 491.214658] env[61770]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 491.215510] env[61770]: WARNING oslo_policy.policy [None req-18bbb090-4b69-4296-82b3-e2243df7d933 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 491.215723] env[61770]: WARNING oslo_policy.policy [None req-18bbb090-4b69-4296-82b3-e2243df7d933 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 491.217144] env[61770]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-18bbb090-4b69-4296-82b3-e2243df7d933 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 491.230686] env[61770]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-18bbb090-4b69-4296-82b3-e2243df7d933 None None] Session token - acquiring [ 491.231476] env[61770]: DEBUG networking_nsxv3.common.locking [None req-18bbb090-4b69-4296-82b3-e2243df7d933 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=61770) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 491.231700] env[61770]: DEBUG oslo_concurrency.lockutils [None req-18bbb090-4b69-4296-82b3-e2243df7d933 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61770) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 491.231830] env[61770]: DEBUG oslo_concurrency.lockutils [None req-18bbb090-4b69-4296-82b3-e2243df7d933 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61770) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 491.232216] env[61770]: DEBUG oslo_concurrency.lockutils [None req-18bbb090-4b69-4296-82b3-e2243df7d933 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61770) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 491.241258] env[61770]: DEBUG oslo_concurrency.lockutils [None req-18bbb090-4b69-4296-82b3-e2243df7d933 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61770) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 491.241515] env[61770]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-18bbb090-4b69-4296-82b3-e2243df7d933 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 491.241663] env[61770]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-18bbb090-4b69-4296-82b3-e2243df7d933 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=61770) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 493.247651] env[61770]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-18bbb090-4b69-4296-82b3-e2243df7d933 None None] Session token - acquiring [ 493.248033] env[61770]: DEBUG networking_nsxv3.common.locking [None req-18bbb090-4b69-4296-82b3-e2243df7d933 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=61770) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 493.248197] env[61770]: DEBUG oslo_concurrency.lockutils [None req-18bbb090-4b69-4296-82b3-e2243df7d933 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61770) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 493.248320] env[61770]: DEBUG oslo_concurrency.lockutils [None req-18bbb090-4b69-4296-82b3-e2243df7d933 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61770) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 493.248625] env[61770]: DEBUG oslo_concurrency.lockutils [None req-18bbb090-4b69-4296-82b3-e2243df7d933 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61770) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 493.258096] env[61770]: DEBUG oslo_concurrency.lockutils [None req-18bbb090-4b69-4296-82b3-e2243df7d933 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61770) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 493.258766] env[61770]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-18bbb090-4b69-4296-82b3-e2243df7d933 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 493.258766] env[61770]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-18bbb090-4b69-4296-82b3-e2243df7d933 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=61770) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 495.261102] env[61770]: CRITICAL neutron [None req-18bbb090-4b69-4296-82b3-e2243df7d933 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 495.261102] env[61770]: ERROR neutron Traceback (most recent call last): [ 495.261102] env[61770]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 495.261102] env[61770]: ERROR neutron sys.exit(main()) [ 495.261102] env[61770]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 495.261102] env[61770]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 495.261102] env[61770]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 495.261102] env[61770]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 495.261102] env[61770]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 495.261102] env[61770]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 495.261102] env[61770]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 495.261102] env[61770]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 495.261102] env[61770]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 495.261102] env[61770]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 495.261102] env[61770]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 495.261102] env[61770]: ERROR neutron response = self.get(path=path, params=params) [ 495.262160] env[61770]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 495.262160] env[61770]: ERROR neutron raise RuntimeError(msg, last_err) [ 495.262160] env[61770]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 495.262160] env[61770]: ERROR neutron [ 497.921247] env[62080]: INFO neutron.common.config [-] Logging enabled! [ 497.921484] env[62080]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 497.921759] env[62080]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=62080) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 497.921980] env[62080]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 497.922067] env[62080]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 497.922923] env[62080]: WARNING oslo_policy.policy [None req-7c83d92c-cff6-47ca-891d-609f7b12bd06 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 497.923138] env[62080]: WARNING oslo_policy.policy [None req-7c83d92c-cff6-47ca-891d-609f7b12bd06 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 497.924445] env[62080]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-7c83d92c-cff6-47ca-891d-609f7b12bd06 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 497.938915] env[62080]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7c83d92c-cff6-47ca-891d-609f7b12bd06 None None] Session token - acquiring [ 497.940202] env[62080]: DEBUG networking_nsxv3.common.locking [None req-7c83d92c-cff6-47ca-891d-609f7b12bd06 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62080) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 497.940202] env[62080]: DEBUG oslo_concurrency.lockutils [None req-7c83d92c-cff6-47ca-891d-609f7b12bd06 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62080) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 497.940400] env[62080]: DEBUG oslo_concurrency.lockutils [None req-7c83d92c-cff6-47ca-891d-609f7b12bd06 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62080) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 497.940856] env[62080]: DEBUG oslo_concurrency.lockutils [None req-7c83d92c-cff6-47ca-891d-609f7b12bd06 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62080) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 497.950409] env[62080]: DEBUG oslo_concurrency.lockutils [None req-7c83d92c-cff6-47ca-891d-609f7b12bd06 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62080) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 497.950729] env[62080]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7c83d92c-cff6-47ca-891d-609f7b12bd06 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 497.950967] env[62080]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7c83d92c-cff6-47ca-891d-609f7b12bd06 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=62080) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 499.959581] env[62080]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7c83d92c-cff6-47ca-891d-609f7b12bd06 None None] Session token - acquiring [ 499.959581] env[62080]: DEBUG networking_nsxv3.common.locking [None req-7c83d92c-cff6-47ca-891d-609f7b12bd06 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62080) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 499.959581] env[62080]: DEBUG oslo_concurrency.lockutils [None req-7c83d92c-cff6-47ca-891d-609f7b12bd06 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62080) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 499.959902] env[62080]: DEBUG oslo_concurrency.lockutils [None req-7c83d92c-cff6-47ca-891d-609f7b12bd06 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62080) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 499.959902] env[62080]: DEBUG oslo_concurrency.lockutils [None req-7c83d92c-cff6-47ca-891d-609f7b12bd06 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62080) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 499.968810] env[62080]: DEBUG oslo_concurrency.lockutils [None req-7c83d92c-cff6-47ca-891d-609f7b12bd06 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62080) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 499.969125] env[62080]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7c83d92c-cff6-47ca-891d-609f7b12bd06 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 499.969327] env[62080]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7c83d92c-cff6-47ca-891d-609f7b12bd06 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=62080) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 501.972275] env[62080]: CRITICAL neutron [None req-7c83d92c-cff6-47ca-891d-609f7b12bd06 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 501.972275] env[62080]: ERROR neutron Traceback (most recent call last): [ 501.972275] env[62080]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 501.972275] env[62080]: ERROR neutron sys.exit(main()) [ 501.972275] env[62080]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 501.972275] env[62080]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 501.972275] env[62080]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 501.972275] env[62080]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 501.972275] env[62080]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 501.972275] env[62080]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 501.972275] env[62080]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 501.972275] env[62080]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 501.972275] env[62080]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 501.972275] env[62080]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 501.972275] env[62080]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 501.972275] env[62080]: ERROR neutron response = self.get(path=path, params=params) [ 501.973000] env[62080]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 501.973000] env[62080]: ERROR neutron raise RuntimeError(msg, last_err) [ 501.973000] env[62080]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 501.973000] env[62080]: ERROR neutron [ 504.883642] env[62740]: INFO neutron.common.config [-] Logging enabled! [ 504.883886] env[62740]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 504.884445] env[62740]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=62740) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 504.884710] env[62740]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 504.884805] env[62740]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 504.886330] env[62740]: WARNING oslo_policy.policy [None req-2a2ef07c-ee14-4e67-bb48-d3ce5a5afee8 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 504.886584] env[62740]: WARNING oslo_policy.policy [None req-2a2ef07c-ee14-4e67-bb48-d3ce5a5afee8 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 504.887832] env[62740]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-2a2ef07c-ee14-4e67-bb48-d3ce5a5afee8 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 504.903497] env[62740]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2a2ef07c-ee14-4e67-bb48-d3ce5a5afee8 None None] Session token - acquiring [ 504.906834] env[62740]: DEBUG networking_nsxv3.common.locking [None req-2a2ef07c-ee14-4e67-bb48-d3ce5a5afee8 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62740) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 504.907182] env[62740]: DEBUG oslo_concurrency.lockutils [None req-2a2ef07c-ee14-4e67-bb48-d3ce5a5afee8 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62740) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 504.907245] env[62740]: DEBUG oslo_concurrency.lockutils [None req-2a2ef07c-ee14-4e67-bb48-d3ce5a5afee8 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62740) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 504.907655] env[62740]: DEBUG oslo_concurrency.lockutils [None req-2a2ef07c-ee14-4e67-bb48-d3ce5a5afee8 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62740) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 504.918320] env[62740]: DEBUG oslo_concurrency.lockutils [None req-2a2ef07c-ee14-4e67-bb48-d3ce5a5afee8 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62740) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 504.918793] env[62740]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2a2ef07c-ee14-4e67-bb48-d3ce5a5afee8 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 504.919080] env[62740]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2a2ef07c-ee14-4e67-bb48-d3ce5a5afee8 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=62740) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 506.925256] env[62740]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2a2ef07c-ee14-4e67-bb48-d3ce5a5afee8 None None] Session token - acquiring [ 506.925635] env[62740]: DEBUG networking_nsxv3.common.locking [None req-2a2ef07c-ee14-4e67-bb48-d3ce5a5afee8 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62740) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 506.925865] env[62740]: DEBUG oslo_concurrency.lockutils [None req-2a2ef07c-ee14-4e67-bb48-d3ce5a5afee8 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62740) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 506.926019] env[62740]: DEBUG oslo_concurrency.lockutils [None req-2a2ef07c-ee14-4e67-bb48-d3ce5a5afee8 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62740) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 506.926343] env[62740]: DEBUG oslo_concurrency.lockutils [None req-2a2ef07c-ee14-4e67-bb48-d3ce5a5afee8 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62740) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 506.939001] env[62740]: DEBUG oslo_concurrency.lockutils [None req-2a2ef07c-ee14-4e67-bb48-d3ce5a5afee8 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62740) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 506.939001] env[62740]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2a2ef07c-ee14-4e67-bb48-d3ce5a5afee8 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 506.939001] env[62740]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2a2ef07c-ee14-4e67-bb48-d3ce5a5afee8 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=62740) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 508.949575] env[62740]: CRITICAL neutron [None req-2a2ef07c-ee14-4e67-bb48-d3ce5a5afee8 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 508.949575] env[62740]: ERROR neutron Traceback (most recent call last): [ 508.949575] env[62740]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 508.949575] env[62740]: ERROR neutron sys.exit(main()) [ 508.949575] env[62740]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 508.949575] env[62740]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 508.949575] env[62740]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 508.949575] env[62740]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 508.949575] env[62740]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 508.949575] env[62740]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 508.949575] env[62740]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 508.949575] env[62740]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 508.949575] env[62740]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 508.949575] env[62740]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 508.949575] env[62740]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 508.949575] env[62740]: ERROR neutron response = self.get(path=path, params=params) [ 508.950681] env[62740]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 508.950681] env[62740]: ERROR neutron raise RuntimeError(msg, last_err) [ 508.950681] env[62740]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 508.950681] env[62740]: ERROR neutron [ 511.596328] env[63089]: INFO neutron.common.config [-] Logging enabled! [ 511.596328] env[63089]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 511.596328] env[63089]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=63089) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 511.596328] env[63089]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 511.596328] env[63089]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 511.597390] env[63089]: WARNING oslo_policy.policy [None req-0b645c47-fea9-4458-beaa-168262ffa97f None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 511.597390] env[63089]: WARNING oslo_policy.policy [None req-0b645c47-fea9-4458-beaa-168262ffa97f None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 511.598415] env[63089]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-0b645c47-fea9-4458-beaa-168262ffa97f None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 511.614096] env[63089]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0b645c47-fea9-4458-beaa-168262ffa97f None None] Session token - acquiring [ 511.615306] env[63089]: DEBUG networking_nsxv3.common.locking [None req-0b645c47-fea9-4458-beaa-168262ffa97f None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=63089) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 511.615407] env[63089]: DEBUG oslo_concurrency.lockutils [None req-0b645c47-fea9-4458-beaa-168262ffa97f None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63089) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 511.615609] env[63089]: DEBUG oslo_concurrency.lockutils [None req-0b645c47-fea9-4458-beaa-168262ffa97f None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63089) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 511.616112] env[63089]: DEBUG oslo_concurrency.lockutils [None req-0b645c47-fea9-4458-beaa-168262ffa97f None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63089) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 511.626013] env[63089]: DEBUG oslo_concurrency.lockutils [None req-0b645c47-fea9-4458-beaa-168262ffa97f None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63089) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 511.626368] env[63089]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0b645c47-fea9-4458-beaa-168262ffa97f None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 511.627728] env[63089]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0b645c47-fea9-4458-beaa-168262ffa97f None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=63089) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 513.633806] env[63089]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0b645c47-fea9-4458-beaa-168262ffa97f None None] Session token - acquiring [ 513.634119] env[63089]: DEBUG networking_nsxv3.common.locking [None req-0b645c47-fea9-4458-beaa-168262ffa97f None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=63089) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 513.634306] env[63089]: DEBUG oslo_concurrency.lockutils [None req-0b645c47-fea9-4458-beaa-168262ffa97f None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63089) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 513.634426] env[63089]: DEBUG oslo_concurrency.lockutils [None req-0b645c47-fea9-4458-beaa-168262ffa97f None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63089) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 513.634736] env[63089]: DEBUG oslo_concurrency.lockutils [None req-0b645c47-fea9-4458-beaa-168262ffa97f None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63089) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 513.644711] env[63089]: DEBUG oslo_concurrency.lockutils [None req-0b645c47-fea9-4458-beaa-168262ffa97f None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63089) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 513.644994] env[63089]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0b645c47-fea9-4458-beaa-168262ffa97f None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 513.645154] env[63089]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0b645c47-fea9-4458-beaa-168262ffa97f None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=63089) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 515.648064] env[63089]: CRITICAL neutron [None req-0b645c47-fea9-4458-beaa-168262ffa97f None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 515.648064] env[63089]: ERROR neutron Traceback (most recent call last): [ 515.648064] env[63089]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 515.648064] env[63089]: ERROR neutron sys.exit(main()) [ 515.648064] env[63089]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 515.648064] env[63089]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 515.648064] env[63089]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 515.648064] env[63089]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 515.648064] env[63089]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 515.648064] env[63089]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 515.648064] env[63089]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 515.648064] env[63089]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 515.648064] env[63089]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 515.648064] env[63089]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 515.648064] env[63089]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 515.648064] env[63089]: ERROR neutron response = self.get(path=path, params=params) [ 515.649330] env[63089]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 515.649330] env[63089]: ERROR neutron raise RuntimeError(msg, last_err) [ 515.649330] env[63089]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 515.649330] env[63089]: ERROR neutron [ 518.569221] env[63368]: INFO neutron.common.config [-] Logging enabled! [ 518.569581] env[63368]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 518.569775] env[63368]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=63368) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 518.570033] env[63368]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 518.570087] env[63368]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 518.570950] env[63368]: WARNING oslo_policy.policy [None req-ca78f0ff-6b93-4366-bb16-5c0272461de1 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 518.571172] env[63368]: WARNING oslo_policy.policy [None req-ca78f0ff-6b93-4366-bb16-5c0272461de1 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 518.572414] env[63368]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-ca78f0ff-6b93-4366-bb16-5c0272461de1 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 518.588180] env[63368]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ca78f0ff-6b93-4366-bb16-5c0272461de1 None None] Session token - acquiring [ 518.589139] env[63368]: DEBUG networking_nsxv3.common.locking [None req-ca78f0ff-6b93-4366-bb16-5c0272461de1 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=63368) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 518.589260] env[63368]: DEBUG oslo_concurrency.lockutils [None req-ca78f0ff-6b93-4366-bb16-5c0272461de1 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63368) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 518.589376] env[63368]: DEBUG oslo_concurrency.lockutils [None req-ca78f0ff-6b93-4366-bb16-5c0272461de1 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63368) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 518.589758] env[63368]: DEBUG oslo_concurrency.lockutils [None req-ca78f0ff-6b93-4366-bb16-5c0272461de1 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63368) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 518.600058] env[63368]: DEBUG oslo_concurrency.lockutils [None req-ca78f0ff-6b93-4366-bb16-5c0272461de1 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63368) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 518.600542] env[63368]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ca78f0ff-6b93-4366-bb16-5c0272461de1 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 518.601168] env[63368]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ca78f0ff-6b93-4366-bb16-5c0272461de1 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=63368) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 520.606796] env[63368]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ca78f0ff-6b93-4366-bb16-5c0272461de1 None None] Session token - acquiring [ 520.608429] env[63368]: DEBUG networking_nsxv3.common.locking [None req-ca78f0ff-6b93-4366-bb16-5c0272461de1 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=63368) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 520.608429] env[63368]: DEBUG oslo_concurrency.lockutils [None req-ca78f0ff-6b93-4366-bb16-5c0272461de1 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63368) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 520.608592] env[63368]: DEBUG oslo_concurrency.lockutils [None req-ca78f0ff-6b93-4366-bb16-5c0272461de1 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63368) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 520.608592] env[63368]: DEBUG oslo_concurrency.lockutils [None req-ca78f0ff-6b93-4366-bb16-5c0272461de1 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63368) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 520.622919] env[63368]: DEBUG oslo_concurrency.lockutils [None req-ca78f0ff-6b93-4366-bb16-5c0272461de1 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63368) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 520.624217] env[63368]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ca78f0ff-6b93-4366-bb16-5c0272461de1 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 520.624369] env[63368]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ca78f0ff-6b93-4366-bb16-5c0272461de1 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=63368) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 522.627300] env[63368]: CRITICAL neutron [None req-ca78f0ff-6b93-4366-bb16-5c0272461de1 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 522.627300] env[63368]: ERROR neutron Traceback (most recent call last): [ 522.627300] env[63368]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 522.627300] env[63368]: ERROR neutron sys.exit(main()) [ 522.627300] env[63368]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 522.627300] env[63368]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 522.627300] env[63368]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 522.627300] env[63368]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 522.627300] env[63368]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 522.627300] env[63368]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 522.627300] env[63368]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 522.627300] env[63368]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 522.627300] env[63368]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 522.627300] env[63368]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 522.627300] env[63368]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 522.627300] env[63368]: ERROR neutron response = self.get(path=path, params=params) [ 522.628099] env[63368]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 522.628099] env[63368]: ERROR neutron raise RuntimeError(msg, last_err) [ 522.628099] env[63368]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 522.628099] env[63368]: ERROR neutron [ 525.565155] env[63862]: INFO neutron.common.config [-] Logging enabled! [ 525.565391] env[63862]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 525.565725] env[63862]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=63862) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 525.566037] env[63862]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 525.566037] env[63862]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 525.566873] env[63862]: WARNING oslo_policy.policy [None req-43c7bec2-2264-4813-af7f-137189572d8a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 525.567094] env[63862]: WARNING oslo_policy.policy [None req-43c7bec2-2264-4813-af7f-137189572d8a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 525.568733] env[63862]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-43c7bec2-2264-4813-af7f-137189572d8a None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 525.583305] env[63862]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-43c7bec2-2264-4813-af7f-137189572d8a None None] Session token - acquiring [ 525.584279] env[63862]: DEBUG networking_nsxv3.common.locking [None req-43c7bec2-2264-4813-af7f-137189572d8a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=63862) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 525.584486] env[63862]: DEBUG oslo_concurrency.lockutils [None req-43c7bec2-2264-4813-af7f-137189572d8a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63862) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 525.584677] env[63862]: DEBUG oslo_concurrency.lockutils [None req-43c7bec2-2264-4813-af7f-137189572d8a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63862) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 525.585122] env[63862]: DEBUG oslo_concurrency.lockutils [None req-43c7bec2-2264-4813-af7f-137189572d8a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63862) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 525.595151] env[63862]: DEBUG oslo_concurrency.lockutils [None req-43c7bec2-2264-4813-af7f-137189572d8a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63862) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 525.595151] env[63862]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-43c7bec2-2264-4813-af7f-137189572d8a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 525.595297] env[63862]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-43c7bec2-2264-4813-af7f-137189572d8a None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=63862) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 527.602236] env[63862]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-43c7bec2-2264-4813-af7f-137189572d8a None None] Session token - acquiring [ 527.602549] env[63862]: DEBUG networking_nsxv3.common.locking [None req-43c7bec2-2264-4813-af7f-137189572d8a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=63862) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 527.602747] env[63862]: DEBUG oslo_concurrency.lockutils [None req-43c7bec2-2264-4813-af7f-137189572d8a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63862) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 527.602905] env[63862]: DEBUG oslo_concurrency.lockutils [None req-43c7bec2-2264-4813-af7f-137189572d8a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63862) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 527.603279] env[63862]: DEBUG oslo_concurrency.lockutils [None req-43c7bec2-2264-4813-af7f-137189572d8a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63862) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 527.614432] env[63862]: DEBUG oslo_concurrency.lockutils [None req-43c7bec2-2264-4813-af7f-137189572d8a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63862) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 527.614759] env[63862]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-43c7bec2-2264-4813-af7f-137189572d8a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 527.614935] env[63862]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-43c7bec2-2264-4813-af7f-137189572d8a None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=63862) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 529.616916] env[63862]: CRITICAL neutron [None req-43c7bec2-2264-4813-af7f-137189572d8a None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 529.616916] env[63862]: ERROR neutron Traceback (most recent call last): [ 529.616916] env[63862]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 529.616916] env[63862]: ERROR neutron sys.exit(main()) [ 529.616916] env[63862]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 529.616916] env[63862]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 529.616916] env[63862]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 529.616916] env[63862]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 529.616916] env[63862]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 529.616916] env[63862]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 529.616916] env[63862]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 529.616916] env[63862]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 529.616916] env[63862]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 529.616916] env[63862]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 529.616916] env[63862]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 529.616916] env[63862]: ERROR neutron response = self.get(path=path, params=params) [ 529.617491] env[63862]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 529.617491] env[63862]: ERROR neutron raise RuntimeError(msg, last_err) [ 529.617491] env[63862]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 529.617491] env[63862]: ERROR neutron [ 532.533554] env[64431]: INFO neutron.common.config [-] Logging enabled! [ 532.533804] env[64431]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 532.534093] env[64431]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=64431) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 532.534330] env[64431]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 532.534417] env[64431]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 532.535300] env[64431]: WARNING oslo_policy.policy [None req-87deaf88-fb8d-43be-8df6-1a26efa5c18a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 532.535587] env[64431]: WARNING oslo_policy.policy [None req-87deaf88-fb8d-43be-8df6-1a26efa5c18a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 532.536996] env[64431]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-87deaf88-fb8d-43be-8df6-1a26efa5c18a None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 532.554912] env[64431]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-87deaf88-fb8d-43be-8df6-1a26efa5c18a None None] Session token - acquiring [ 532.555762] env[64431]: DEBUG networking_nsxv3.common.locking [None req-87deaf88-fb8d-43be-8df6-1a26efa5c18a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=64431) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 532.555987] env[64431]: DEBUG oslo_concurrency.lockutils [None req-87deaf88-fb8d-43be-8df6-1a26efa5c18a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64431) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 532.556160] env[64431]: DEBUG oslo_concurrency.lockutils [None req-87deaf88-fb8d-43be-8df6-1a26efa5c18a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64431) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 532.556623] env[64431]: DEBUG oslo_concurrency.lockutils [None req-87deaf88-fb8d-43be-8df6-1a26efa5c18a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64431) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 532.576937] env[64431]: DEBUG oslo_concurrency.lockutils [None req-87deaf88-fb8d-43be-8df6-1a26efa5c18a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64431) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 532.576937] env[64431]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-87deaf88-fb8d-43be-8df6-1a26efa5c18a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 532.576937] env[64431]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-87deaf88-fb8d-43be-8df6-1a26efa5c18a None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=64431) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 534.582132] env[64431]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-87deaf88-fb8d-43be-8df6-1a26efa5c18a None None] Session token - acquiring [ 534.584179] env[64431]: DEBUG networking_nsxv3.common.locking [None req-87deaf88-fb8d-43be-8df6-1a26efa5c18a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=64431) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 534.584179] env[64431]: DEBUG oslo_concurrency.lockutils [None req-87deaf88-fb8d-43be-8df6-1a26efa5c18a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64431) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 534.584317] env[64431]: DEBUG oslo_concurrency.lockutils [None req-87deaf88-fb8d-43be-8df6-1a26efa5c18a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64431) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 534.584317] env[64431]: DEBUG oslo_concurrency.lockutils [None req-87deaf88-fb8d-43be-8df6-1a26efa5c18a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64431) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 534.594393] env[64431]: DEBUG oslo_concurrency.lockutils [None req-87deaf88-fb8d-43be-8df6-1a26efa5c18a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64431) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 534.594687] env[64431]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-87deaf88-fb8d-43be-8df6-1a26efa5c18a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 534.594845] env[64431]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-87deaf88-fb8d-43be-8df6-1a26efa5c18a None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=64431) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 536.596883] env[64431]: CRITICAL neutron [None req-87deaf88-fb8d-43be-8df6-1a26efa5c18a None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 536.596883] env[64431]: ERROR neutron Traceback (most recent call last): [ 536.596883] env[64431]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 536.596883] env[64431]: ERROR neutron sys.exit(main()) [ 536.596883] env[64431]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 536.596883] env[64431]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 536.596883] env[64431]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 536.596883] env[64431]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 536.596883] env[64431]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 536.596883] env[64431]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 536.596883] env[64431]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 536.596883] env[64431]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 536.596883] env[64431]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 536.596883] env[64431]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 536.596883] env[64431]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 536.596883] env[64431]: ERROR neutron response = self.get(path=path, params=params) [ 536.597589] env[64431]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 536.597589] env[64431]: ERROR neutron raise RuntimeError(msg, last_err) [ 536.597589] env[64431]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 536.597589] env[64431]: ERROR neutron [ 539.654171] env[64898]: INFO neutron.common.config [-] Logging enabled! [ 539.654435] env[64898]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 539.654713] env[64898]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=64898) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 539.654958] env[64898]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 539.655045] env[64898]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 539.655898] env[64898]: WARNING oslo_policy.policy [None req-ee89a24b-3dee-46e1-a4b8-9c710d588a1d None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 539.656128] env[64898]: WARNING oslo_policy.policy [None req-ee89a24b-3dee-46e1-a4b8-9c710d588a1d None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 539.657354] env[64898]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-ee89a24b-3dee-46e1-a4b8-9c710d588a1d None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 539.672358] env[64898]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ee89a24b-3dee-46e1-a4b8-9c710d588a1d None None] Session token - acquiring [ 539.673153] env[64898]: DEBUG networking_nsxv3.common.locking [None req-ee89a24b-3dee-46e1-a4b8-9c710d588a1d None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=64898) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 539.673355] env[64898]: DEBUG oslo_concurrency.lockutils [None req-ee89a24b-3dee-46e1-a4b8-9c710d588a1d None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64898) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 539.673479] env[64898]: DEBUG oslo_concurrency.lockutils [None req-ee89a24b-3dee-46e1-a4b8-9c710d588a1d None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64898) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 539.673890] env[64898]: DEBUG oslo_concurrency.lockutils [None req-ee89a24b-3dee-46e1-a4b8-9c710d588a1d None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64898) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 539.683919] env[64898]: DEBUG oslo_concurrency.lockutils [None req-ee89a24b-3dee-46e1-a4b8-9c710d588a1d None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64898) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 539.684312] env[64898]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ee89a24b-3dee-46e1-a4b8-9c710d588a1d None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 539.684368] env[64898]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ee89a24b-3dee-46e1-a4b8-9c710d588a1d None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=64898) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 541.690453] env[64898]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ee89a24b-3dee-46e1-a4b8-9c710d588a1d None None] Session token - acquiring [ 541.690794] env[64898]: DEBUG networking_nsxv3.common.locking [None req-ee89a24b-3dee-46e1-a4b8-9c710d588a1d None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=64898) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 541.690990] env[64898]: DEBUG oslo_concurrency.lockutils [None req-ee89a24b-3dee-46e1-a4b8-9c710d588a1d None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64898) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 541.691130] env[64898]: DEBUG oslo_concurrency.lockutils [None req-ee89a24b-3dee-46e1-a4b8-9c710d588a1d None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64898) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 541.691406] env[64898]: DEBUG oslo_concurrency.lockutils [None req-ee89a24b-3dee-46e1-a4b8-9c710d588a1d None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64898) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 541.700839] env[64898]: DEBUG oslo_concurrency.lockutils [None req-ee89a24b-3dee-46e1-a4b8-9c710d588a1d None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64898) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 541.701095] env[64898]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ee89a24b-3dee-46e1-a4b8-9c710d588a1d None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 541.701235] env[64898]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ee89a24b-3dee-46e1-a4b8-9c710d588a1d None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=64898) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 543.705814] env[64898]: CRITICAL neutron [None req-ee89a24b-3dee-46e1-a4b8-9c710d588a1d None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 543.705814] env[64898]: ERROR neutron Traceback (most recent call last): [ 543.705814] env[64898]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 543.705814] env[64898]: ERROR neutron sys.exit(main()) [ 543.705814] env[64898]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 543.705814] env[64898]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 543.705814] env[64898]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 543.705814] env[64898]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 543.705814] env[64898]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 543.705814] env[64898]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 543.705814] env[64898]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 543.705814] env[64898]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 543.705814] env[64898]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 543.705814] env[64898]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 543.705814] env[64898]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 543.705814] env[64898]: ERROR neutron response = self.get(path=path, params=params) [ 543.706667] env[64898]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 543.706667] env[64898]: ERROR neutron raise RuntimeError(msg, last_err) [ 543.706667] env[64898]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 543.706667] env[64898]: ERROR neutron [ 546.658515] env[65265]: INFO neutron.common.config [-] Logging enabled! [ 546.658796] env[65265]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 546.658971] env[65265]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=65265) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 546.659206] env[65265]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 546.659296] env[65265]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 546.660215] env[65265]: WARNING oslo_policy.policy [None req-a012bbf1-2347-47e6-9d32-2a14c6ae5d98 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 546.660431] env[65265]: WARNING oslo_policy.policy [None req-a012bbf1-2347-47e6-9d32-2a14c6ae5d98 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 546.661739] env[65265]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-a012bbf1-2347-47e6-9d32-2a14c6ae5d98 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 546.676666] env[65265]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a012bbf1-2347-47e6-9d32-2a14c6ae5d98 None None] Session token - acquiring [ 546.677606] env[65265]: DEBUG networking_nsxv3.common.locking [None req-a012bbf1-2347-47e6-9d32-2a14c6ae5d98 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=65265) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 546.677815] env[65265]: DEBUG oslo_concurrency.lockutils [None req-a012bbf1-2347-47e6-9d32-2a14c6ae5d98 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65265) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 546.677938] env[65265]: DEBUG oslo_concurrency.lockutils [None req-a012bbf1-2347-47e6-9d32-2a14c6ae5d98 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65265) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 546.678323] env[65265]: DEBUG oslo_concurrency.lockutils [None req-a012bbf1-2347-47e6-9d32-2a14c6ae5d98 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65265) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 546.700061] env[65265]: DEBUG oslo_concurrency.lockutils [None req-a012bbf1-2347-47e6-9d32-2a14c6ae5d98 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65265) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 546.700338] env[65265]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a012bbf1-2347-47e6-9d32-2a14c6ae5d98 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 546.700480] env[65265]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a012bbf1-2347-47e6-9d32-2a14c6ae5d98 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=65265) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 548.708760] env[65265]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a012bbf1-2347-47e6-9d32-2a14c6ae5d98 None None] Session token - acquiring [ 548.708760] env[65265]: DEBUG networking_nsxv3.common.locking [None req-a012bbf1-2347-47e6-9d32-2a14c6ae5d98 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=65265) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 548.708760] env[65265]: DEBUG oslo_concurrency.lockutils [None req-a012bbf1-2347-47e6-9d32-2a14c6ae5d98 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65265) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 548.709336] env[65265]: DEBUG oslo_concurrency.lockutils [None req-a012bbf1-2347-47e6-9d32-2a14c6ae5d98 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65265) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 548.709336] env[65265]: DEBUG oslo_concurrency.lockutils [None req-a012bbf1-2347-47e6-9d32-2a14c6ae5d98 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65265) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 548.717467] env[65265]: DEBUG oslo_concurrency.lockutils [None req-a012bbf1-2347-47e6-9d32-2a14c6ae5d98 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65265) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 548.717902] env[65265]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a012bbf1-2347-47e6-9d32-2a14c6ae5d98 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 548.717902] env[65265]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a012bbf1-2347-47e6-9d32-2a14c6ae5d98 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=65265) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 550.720781] env[65265]: CRITICAL neutron [None req-a012bbf1-2347-47e6-9d32-2a14c6ae5d98 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 550.720781] env[65265]: ERROR neutron Traceback (most recent call last): [ 550.720781] env[65265]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 550.720781] env[65265]: ERROR neutron sys.exit(main()) [ 550.720781] env[65265]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 550.720781] env[65265]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 550.720781] env[65265]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 550.720781] env[65265]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 550.720781] env[65265]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 550.720781] env[65265]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 550.720781] env[65265]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 550.720781] env[65265]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 550.720781] env[65265]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 550.720781] env[65265]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 550.720781] env[65265]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 550.720781] env[65265]: ERROR neutron response = self.get(path=path, params=params) [ 550.721643] env[65265]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 550.721643] env[65265]: ERROR neutron raise RuntimeError(msg, last_err) [ 550.721643] env[65265]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 550.721643] env[65265]: ERROR neutron [ 553.649397] env[65889]: INFO neutron.common.config [-] Logging enabled! [ 553.649656] env[65889]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 553.649971] env[65889]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=65889) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 553.650209] env[65889]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 553.650300] env[65889]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 553.651211] env[65889]: WARNING oslo_policy.policy [None req-76966ad6-453c-467b-a25c-47ab1c572a46 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 553.651421] env[65889]: WARNING oslo_policy.policy [None req-76966ad6-453c-467b-a25c-47ab1c572a46 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 553.660239] env[65889]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-76966ad6-453c-467b-a25c-47ab1c572a46 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 553.680250] env[65889]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-76966ad6-453c-467b-a25c-47ab1c572a46 None None] Session token - acquiring [ 553.681120] env[65889]: DEBUG networking_nsxv3.common.locking [None req-76966ad6-453c-467b-a25c-47ab1c572a46 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=65889) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 553.681333] env[65889]: DEBUG oslo_concurrency.lockutils [None req-76966ad6-453c-467b-a25c-47ab1c572a46 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65889) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 553.681462] env[65889]: DEBUG oslo_concurrency.lockutils [None req-76966ad6-453c-467b-a25c-47ab1c572a46 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65889) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 553.681845] env[65889]: DEBUG oslo_concurrency.lockutils [None req-76966ad6-453c-467b-a25c-47ab1c572a46 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65889) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 553.694144] env[65889]: DEBUG oslo_concurrency.lockutils [None req-76966ad6-453c-467b-a25c-47ab1c572a46 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65889) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 553.694425] env[65889]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-76966ad6-453c-467b-a25c-47ab1c572a46 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 553.694569] env[65889]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-76966ad6-453c-467b-a25c-47ab1c572a46 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=65889) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 555.703224] env[65889]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-76966ad6-453c-467b-a25c-47ab1c572a46 None None] Session token - acquiring [ 555.703554] env[65889]: DEBUG networking_nsxv3.common.locking [None req-76966ad6-453c-467b-a25c-47ab1c572a46 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=65889) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 555.703803] env[65889]: DEBUG oslo_concurrency.lockutils [None req-76966ad6-453c-467b-a25c-47ab1c572a46 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65889) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 555.703890] env[65889]: DEBUG oslo_concurrency.lockutils [None req-76966ad6-453c-467b-a25c-47ab1c572a46 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65889) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 555.705393] env[65889]: DEBUG oslo_concurrency.lockutils [None req-76966ad6-453c-467b-a25c-47ab1c572a46 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65889) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 555.715069] env[65889]: DEBUG oslo_concurrency.lockutils [None req-76966ad6-453c-467b-a25c-47ab1c572a46 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65889) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 555.715336] env[65889]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-76966ad6-453c-467b-a25c-47ab1c572a46 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 555.715478] env[65889]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-76966ad6-453c-467b-a25c-47ab1c572a46 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=65889) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 557.716906] env[65889]: CRITICAL neutron [None req-76966ad6-453c-467b-a25c-47ab1c572a46 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 557.716906] env[65889]: ERROR neutron Traceback (most recent call last): [ 557.716906] env[65889]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 557.716906] env[65889]: ERROR neutron sys.exit(main()) [ 557.716906] env[65889]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 557.716906] env[65889]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 557.716906] env[65889]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 557.716906] env[65889]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 557.716906] env[65889]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 557.716906] env[65889]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 557.716906] env[65889]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 557.716906] env[65889]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 557.716906] env[65889]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 557.716906] env[65889]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 557.716906] env[65889]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 557.716906] env[65889]: ERROR neutron response = self.get(path=path, params=params) [ 557.717804] env[65889]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 557.717804] env[65889]: ERROR neutron raise RuntimeError(msg, last_err) [ 557.717804] env[65889]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 557.717804] env[65889]: ERROR neutron [ 560.409811] env[66684]: INFO neutron.common.config [-] Logging enabled! [ 560.410055] env[66684]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 560.410387] env[66684]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=66684) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 560.410624] env[66684]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 560.410719] env[66684]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 560.411613] env[66684]: WARNING oslo_policy.policy [None req-f1d396d2-5b4d-4c5c-8077-529212b5619c None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 560.411843] env[66684]: WARNING oslo_policy.policy [None req-f1d396d2-5b4d-4c5c-8077-529212b5619c None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 560.417983] env[66684]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-f1d396d2-5b4d-4c5c-8077-529212b5619c None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 560.428190] env[66684]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f1d396d2-5b4d-4c5c-8077-529212b5619c None None] Session token - acquiring [ 560.429090] env[66684]: DEBUG networking_nsxv3.common.locking [None req-f1d396d2-5b4d-4c5c-8077-529212b5619c None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=66684) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 560.429331] env[66684]: DEBUG oslo_concurrency.lockutils [None req-f1d396d2-5b4d-4c5c-8077-529212b5619c None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66684) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 560.429466] env[66684]: DEBUG oslo_concurrency.lockutils [None req-f1d396d2-5b4d-4c5c-8077-529212b5619c None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66684) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 560.429865] env[66684]: DEBUG oslo_concurrency.lockutils [None req-f1d396d2-5b4d-4c5c-8077-529212b5619c None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66684) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 560.439247] env[66684]: DEBUG oslo_concurrency.lockutils [None req-f1d396d2-5b4d-4c5c-8077-529212b5619c None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66684) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 560.439546] env[66684]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f1d396d2-5b4d-4c5c-8077-529212b5619c None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 560.439719] env[66684]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f1d396d2-5b4d-4c5c-8077-529212b5619c None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=66684) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 562.445969] env[66684]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f1d396d2-5b4d-4c5c-8077-529212b5619c None None] Session token - acquiring [ 562.446241] env[66684]: DEBUG networking_nsxv3.common.locking [None req-f1d396d2-5b4d-4c5c-8077-529212b5619c None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=66684) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 562.446446] env[66684]: DEBUG oslo_concurrency.lockutils [None req-f1d396d2-5b4d-4c5c-8077-529212b5619c None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66684) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 562.446576] env[66684]: DEBUG oslo_concurrency.lockutils [None req-f1d396d2-5b4d-4c5c-8077-529212b5619c None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66684) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 562.446733] env[66684]: DEBUG oslo_concurrency.lockutils [None req-f1d396d2-5b4d-4c5c-8077-529212b5619c None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66684) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 562.455952] env[66684]: DEBUG oslo_concurrency.lockutils [None req-f1d396d2-5b4d-4c5c-8077-529212b5619c None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66684) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 562.456256] env[66684]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f1d396d2-5b4d-4c5c-8077-529212b5619c None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 562.456400] env[66684]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f1d396d2-5b4d-4c5c-8077-529212b5619c None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=66684) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 564.459535] env[66684]: CRITICAL neutron [None req-f1d396d2-5b4d-4c5c-8077-529212b5619c None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 564.459535] env[66684]: ERROR neutron Traceback (most recent call last): [ 564.459535] env[66684]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 564.459535] env[66684]: ERROR neutron sys.exit(main()) [ 564.459535] env[66684]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 564.459535] env[66684]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 564.459535] env[66684]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 564.459535] env[66684]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 564.459535] env[66684]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 564.459535] env[66684]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 564.459535] env[66684]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 564.459535] env[66684]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 564.459535] env[66684]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 564.459535] env[66684]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 564.459535] env[66684]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 564.459535] env[66684]: ERROR neutron response = self.get(path=path, params=params) [ 564.460319] env[66684]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 564.460319] env[66684]: ERROR neutron raise RuntimeError(msg, last_err) [ 564.460319] env[66684]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 564.460319] env[66684]: ERROR neutron [ 567.227383] env[66936]: INFO neutron.common.config [-] Logging enabled! [ 567.227651] env[66936]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 567.227925] env[66936]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=66936) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 567.228207] env[66936]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 567.228299] env[66936]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 567.229160] env[66936]: WARNING oslo_policy.policy [None req-5e3b01b8-c0fc-4ae8-a74a-2b3df8d8db0b None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 567.229383] env[66936]: WARNING oslo_policy.policy [None req-5e3b01b8-c0fc-4ae8-a74a-2b3df8d8db0b None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 567.230652] env[66936]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-5e3b01b8-c0fc-4ae8-a74a-2b3df8d8db0b None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 567.244551] env[66936]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5e3b01b8-c0fc-4ae8-a74a-2b3df8d8db0b None None] Session token - acquiring [ 567.245375] env[66936]: DEBUG networking_nsxv3.common.locking [None req-5e3b01b8-c0fc-4ae8-a74a-2b3df8d8db0b None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=66936) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 567.245581] env[66936]: DEBUG oslo_concurrency.lockutils [None req-5e3b01b8-c0fc-4ae8-a74a-2b3df8d8db0b None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66936) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 567.245710] env[66936]: DEBUG oslo_concurrency.lockutils [None req-5e3b01b8-c0fc-4ae8-a74a-2b3df8d8db0b None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66936) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 567.246118] env[66936]: DEBUG oslo_concurrency.lockutils [None req-5e3b01b8-c0fc-4ae8-a74a-2b3df8d8db0b None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66936) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 567.255388] env[66936]: DEBUG oslo_concurrency.lockutils [None req-5e3b01b8-c0fc-4ae8-a74a-2b3df8d8db0b None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66936) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 567.255612] env[66936]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5e3b01b8-c0fc-4ae8-a74a-2b3df8d8db0b None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 567.255858] env[66936]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5e3b01b8-c0fc-4ae8-a74a-2b3df8d8db0b None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=66936) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 569.264506] env[66936]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5e3b01b8-c0fc-4ae8-a74a-2b3df8d8db0b None None] Session token - acquiring [ 569.264506] env[66936]: DEBUG networking_nsxv3.common.locking [None req-5e3b01b8-c0fc-4ae8-a74a-2b3df8d8db0b None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=66936) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 569.264506] env[66936]: DEBUG oslo_concurrency.lockutils [None req-5e3b01b8-c0fc-4ae8-a74a-2b3df8d8db0b None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66936) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 569.264856] env[66936]: DEBUG oslo_concurrency.lockutils [None req-5e3b01b8-c0fc-4ae8-a74a-2b3df8d8db0b None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66936) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 569.264856] env[66936]: DEBUG oslo_concurrency.lockutils [None req-5e3b01b8-c0fc-4ae8-a74a-2b3df8d8db0b None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66936) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 569.276709] env[66936]: DEBUG oslo_concurrency.lockutils [None req-5e3b01b8-c0fc-4ae8-a74a-2b3df8d8db0b None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66936) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 569.277693] env[66936]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5e3b01b8-c0fc-4ae8-a74a-2b3df8d8db0b None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 569.278829] env[66936]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5e3b01b8-c0fc-4ae8-a74a-2b3df8d8db0b None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=66936) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 571.280981] env[66936]: CRITICAL neutron [None req-5e3b01b8-c0fc-4ae8-a74a-2b3df8d8db0b None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 571.280981] env[66936]: ERROR neutron Traceback (most recent call last): [ 571.280981] env[66936]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 571.280981] env[66936]: ERROR neutron sys.exit(main()) [ 571.280981] env[66936]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 571.280981] env[66936]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 571.280981] env[66936]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 571.280981] env[66936]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 571.280981] env[66936]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 571.280981] env[66936]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 571.280981] env[66936]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 571.280981] env[66936]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 571.280981] env[66936]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 571.280981] env[66936]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 571.280981] env[66936]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 571.280981] env[66936]: ERROR neutron response = self.get(path=path, params=params) [ 571.281779] env[66936]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 571.281779] env[66936]: ERROR neutron raise RuntimeError(msg, last_err) [ 571.281779] env[66936]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 571.281779] env[66936]: ERROR neutron [ 574.040347] env[67494]: INFO neutron.common.config [-] Logging enabled! [ 574.040603] env[67494]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 574.040899] env[67494]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=67494) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 574.041122] env[67494]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 574.041209] env[67494]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 574.042052] env[67494]: WARNING oslo_policy.policy [None req-1ad72576-c9b0-44d0-a6b2-0aba703cdf4f None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 574.042259] env[67494]: WARNING oslo_policy.policy [None req-1ad72576-c9b0-44d0-a6b2-0aba703cdf4f None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 574.044467] env[67494]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-1ad72576-c9b0-44d0-a6b2-0aba703cdf4f None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 574.059020] env[67494]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1ad72576-c9b0-44d0-a6b2-0aba703cdf4f None None] Session token - acquiring [ 574.059857] env[67494]: DEBUG networking_nsxv3.common.locking [None req-1ad72576-c9b0-44d0-a6b2-0aba703cdf4f None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=67494) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 574.060084] env[67494]: DEBUG oslo_concurrency.lockutils [None req-1ad72576-c9b0-44d0-a6b2-0aba703cdf4f None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67494) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 574.060210] env[67494]: DEBUG oslo_concurrency.lockutils [None req-1ad72576-c9b0-44d0-a6b2-0aba703cdf4f None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67494) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 574.060583] env[67494]: DEBUG oslo_concurrency.lockutils [None req-1ad72576-c9b0-44d0-a6b2-0aba703cdf4f None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67494) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 574.069902] env[67494]: DEBUG oslo_concurrency.lockutils [None req-1ad72576-c9b0-44d0-a6b2-0aba703cdf4f None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67494) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 574.070150] env[67494]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1ad72576-c9b0-44d0-a6b2-0aba703cdf4f None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 574.070286] env[67494]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1ad72576-c9b0-44d0-a6b2-0aba703cdf4f None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=67494) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 576.077125] env[67494]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1ad72576-c9b0-44d0-a6b2-0aba703cdf4f None None] Session token - acquiring [ 576.077429] env[67494]: DEBUG networking_nsxv3.common.locking [None req-1ad72576-c9b0-44d0-a6b2-0aba703cdf4f None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=67494) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 576.077578] env[67494]: DEBUG oslo_concurrency.lockutils [None req-1ad72576-c9b0-44d0-a6b2-0aba703cdf4f None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67494) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 576.077694] env[67494]: DEBUG oslo_concurrency.lockutils [None req-1ad72576-c9b0-44d0-a6b2-0aba703cdf4f None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67494) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 576.077972] env[67494]: DEBUG oslo_concurrency.lockutils [None req-1ad72576-c9b0-44d0-a6b2-0aba703cdf4f None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67494) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 576.088621] env[67494]: DEBUG oslo_concurrency.lockutils [None req-1ad72576-c9b0-44d0-a6b2-0aba703cdf4f None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67494) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 576.088873] env[67494]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1ad72576-c9b0-44d0-a6b2-0aba703cdf4f None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 576.089013] env[67494]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1ad72576-c9b0-44d0-a6b2-0aba703cdf4f None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=67494) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 578.099569] env[67494]: CRITICAL neutron [None req-1ad72576-c9b0-44d0-a6b2-0aba703cdf4f None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 578.099569] env[67494]: ERROR neutron Traceback (most recent call last): [ 578.099569] env[67494]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 578.099569] env[67494]: ERROR neutron sys.exit(main()) [ 578.099569] env[67494]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 578.099569] env[67494]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 578.099569] env[67494]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 578.099569] env[67494]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 578.099569] env[67494]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 578.099569] env[67494]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 578.099569] env[67494]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 578.099569] env[67494]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 578.099569] env[67494]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 578.099569] env[67494]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 578.099569] env[67494]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 578.099569] env[67494]: ERROR neutron response = self.get(path=path, params=params) [ 578.100327] env[67494]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 578.100327] env[67494]: ERROR neutron raise RuntimeError(msg, last_err) [ 578.100327] env[67494]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 578.100327] env[67494]: ERROR neutron [ 581.126103] env[67699]: INFO neutron.common.config [-] Logging enabled! [ 581.126349] env[67699]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 581.126648] env[67699]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=67699) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 581.126874] env[67699]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 581.126964] env[67699]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 581.127811] env[67699]: WARNING oslo_policy.policy [None req-63e967bb-d311-42e5-b400-430ca32aa4e8 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 581.128038] env[67699]: WARNING oslo_policy.policy [None req-63e967bb-d311-42e5-b400-430ca32aa4e8 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 581.129304] env[67699]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-63e967bb-d311-42e5-b400-430ca32aa4e8 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 581.143614] env[67699]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63e967bb-d311-42e5-b400-430ca32aa4e8 None None] Session token - acquiring [ 581.144483] env[67699]: DEBUG networking_nsxv3.common.locking [None req-63e967bb-d311-42e5-b400-430ca32aa4e8 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=67699) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 581.144685] env[67699]: DEBUG oslo_concurrency.lockutils [None req-63e967bb-d311-42e5-b400-430ca32aa4e8 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67699) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 581.144810] env[67699]: DEBUG oslo_concurrency.lockutils [None req-63e967bb-d311-42e5-b400-430ca32aa4e8 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67699) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 581.145177] env[67699]: DEBUG oslo_concurrency.lockutils [None req-63e967bb-d311-42e5-b400-430ca32aa4e8 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67699) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 581.156341] env[67699]: DEBUG oslo_concurrency.lockutils [None req-63e967bb-d311-42e5-b400-430ca32aa4e8 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67699) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 581.156620] env[67699]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63e967bb-d311-42e5-b400-430ca32aa4e8 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 581.156743] env[67699]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63e967bb-d311-42e5-b400-430ca32aa4e8 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=67699) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 583.163226] env[67699]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63e967bb-d311-42e5-b400-430ca32aa4e8 None None] Session token - acquiring [ 583.163521] env[67699]: DEBUG networking_nsxv3.common.locking [None req-63e967bb-d311-42e5-b400-430ca32aa4e8 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=67699) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 583.163734] env[67699]: DEBUG oslo_concurrency.lockutils [None req-63e967bb-d311-42e5-b400-430ca32aa4e8 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67699) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 583.163853] env[67699]: DEBUG oslo_concurrency.lockutils [None req-63e967bb-d311-42e5-b400-430ca32aa4e8 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67699) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 583.164140] env[67699]: DEBUG oslo_concurrency.lockutils [None req-63e967bb-d311-42e5-b400-430ca32aa4e8 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67699) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 583.173505] env[67699]: DEBUG oslo_concurrency.lockutils [None req-63e967bb-d311-42e5-b400-430ca32aa4e8 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67699) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 583.173783] env[67699]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63e967bb-d311-42e5-b400-430ca32aa4e8 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 583.173913] env[67699]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63e967bb-d311-42e5-b400-430ca32aa4e8 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=67699) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 585.177638] env[67699]: CRITICAL neutron [None req-63e967bb-d311-42e5-b400-430ca32aa4e8 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 585.177638] env[67699]: ERROR neutron Traceback (most recent call last): [ 585.177638] env[67699]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 585.177638] env[67699]: ERROR neutron sys.exit(main()) [ 585.177638] env[67699]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 585.177638] env[67699]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 585.177638] env[67699]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 585.177638] env[67699]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 585.177638] env[67699]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 585.177638] env[67699]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 585.177638] env[67699]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 585.177638] env[67699]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 585.177638] env[67699]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 585.177638] env[67699]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 585.177638] env[67699]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 585.177638] env[67699]: ERROR neutron response = self.get(path=path, params=params) [ 585.178279] env[67699]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 585.178279] env[67699]: ERROR neutron raise RuntimeError(msg, last_err) [ 585.178279] env[67699]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 585.178279] env[67699]: ERROR neutron [ 587.939346] env[68038]: INFO neutron.common.config [-] Logging enabled! [ 587.939346] env[68038]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 587.939791] env[68038]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=68038) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 587.940046] env[68038]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 587.940130] env[68038]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 587.940976] env[68038]: WARNING oslo_policy.policy [None req-577c3fe4-c2cf-4e74-bd4a-ff85d15f9044 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 587.941184] env[68038]: WARNING oslo_policy.policy [None req-577c3fe4-c2cf-4e74-bd4a-ff85d15f9044 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 587.942391] env[68038]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-577c3fe4-c2cf-4e74-bd4a-ff85d15f9044 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 587.957286] env[68038]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-577c3fe4-c2cf-4e74-bd4a-ff85d15f9044 None None] Session token - acquiring [ 587.958097] env[68038]: DEBUG networking_nsxv3.common.locking [None req-577c3fe4-c2cf-4e74-bd4a-ff85d15f9044 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68038) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 587.958314] env[68038]: DEBUG oslo_concurrency.lockutils [None req-577c3fe4-c2cf-4e74-bd4a-ff85d15f9044 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68038) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 587.958426] env[68038]: DEBUG oslo_concurrency.lockutils [None req-577c3fe4-c2cf-4e74-bd4a-ff85d15f9044 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68038) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 587.959303] env[68038]: DEBUG oslo_concurrency.lockutils [None req-577c3fe4-c2cf-4e74-bd4a-ff85d15f9044 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68038) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 587.969772] env[68038]: DEBUG oslo_concurrency.lockutils [None req-577c3fe4-c2cf-4e74-bd4a-ff85d15f9044 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68038) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 587.970335] env[68038]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-577c3fe4-c2cf-4e74-bd4a-ff85d15f9044 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 587.970335] env[68038]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-577c3fe4-c2cf-4e74-bd4a-ff85d15f9044 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=68038) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 589.976365] env[68038]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-577c3fe4-c2cf-4e74-bd4a-ff85d15f9044 None None] Session token - acquiring [ 589.976684] env[68038]: DEBUG networking_nsxv3.common.locking [None req-577c3fe4-c2cf-4e74-bd4a-ff85d15f9044 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68038) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 589.976881] env[68038]: DEBUG oslo_concurrency.lockutils [None req-577c3fe4-c2cf-4e74-bd4a-ff85d15f9044 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68038) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 589.977007] env[68038]: DEBUG oslo_concurrency.lockutils [None req-577c3fe4-c2cf-4e74-bd4a-ff85d15f9044 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68038) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 589.977293] env[68038]: DEBUG oslo_concurrency.lockutils [None req-577c3fe4-c2cf-4e74-bd4a-ff85d15f9044 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68038) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 589.988029] env[68038]: DEBUG oslo_concurrency.lockutils [None req-577c3fe4-c2cf-4e74-bd4a-ff85d15f9044 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68038) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 589.988029] env[68038]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-577c3fe4-c2cf-4e74-bd4a-ff85d15f9044 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 589.988029] env[68038]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-577c3fe4-c2cf-4e74-bd4a-ff85d15f9044 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=68038) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 591.990596] env[68038]: CRITICAL neutron [None req-577c3fe4-c2cf-4e74-bd4a-ff85d15f9044 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 591.990596] env[68038]: ERROR neutron Traceback (most recent call last): [ 591.990596] env[68038]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 591.990596] env[68038]: ERROR neutron sys.exit(main()) [ 591.990596] env[68038]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 591.990596] env[68038]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 591.990596] env[68038]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 591.990596] env[68038]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 591.990596] env[68038]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 591.990596] env[68038]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 591.990596] env[68038]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 591.990596] env[68038]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 591.990596] env[68038]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 591.990596] env[68038]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 591.990596] env[68038]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 591.990596] env[68038]: ERROR neutron response = self.get(path=path, params=params) [ 591.991307] env[68038]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 591.991307] env[68038]: ERROR neutron raise RuntimeError(msg, last_err) [ 591.991307] env[68038]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 591.991307] env[68038]: ERROR neutron [ 594.750920] env[68280]: INFO neutron.common.config [-] Logging enabled! [ 594.751230] env[68280]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 594.751475] env[68280]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=68280) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 594.751730] env[68280]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 594.751819] env[68280]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 594.752680] env[68280]: WARNING oslo_policy.policy [None req-f56f28aa-2a0b-4f8e-9f4c-13eec5040a5e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 594.752896] env[68280]: WARNING oslo_policy.policy [None req-f56f28aa-2a0b-4f8e-9f4c-13eec5040a5e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 594.754166] env[68280]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-f56f28aa-2a0b-4f8e-9f4c-13eec5040a5e None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 594.768612] env[68280]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f56f28aa-2a0b-4f8e-9f4c-13eec5040a5e None None] Session token - acquiring [ 594.769513] env[68280]: DEBUG networking_nsxv3.common.locking [None req-f56f28aa-2a0b-4f8e-9f4c-13eec5040a5e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68280) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 594.769724] env[68280]: DEBUG oslo_concurrency.lockutils [None req-f56f28aa-2a0b-4f8e-9f4c-13eec5040a5e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68280) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 594.769857] env[68280]: DEBUG oslo_concurrency.lockutils [None req-f56f28aa-2a0b-4f8e-9f4c-13eec5040a5e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68280) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 594.770233] env[68280]: DEBUG oslo_concurrency.lockutils [None req-f56f28aa-2a0b-4f8e-9f4c-13eec5040a5e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68280) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 594.779930] env[68280]: DEBUG oslo_concurrency.lockutils [None req-f56f28aa-2a0b-4f8e-9f4c-13eec5040a5e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68280) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 594.780205] env[68280]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f56f28aa-2a0b-4f8e-9f4c-13eec5040a5e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 594.780345] env[68280]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f56f28aa-2a0b-4f8e-9f4c-13eec5040a5e None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=68280) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 596.786281] env[68280]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f56f28aa-2a0b-4f8e-9f4c-13eec5040a5e None None] Session token - acquiring [ 596.786629] env[68280]: DEBUG networking_nsxv3.common.locking [None req-f56f28aa-2a0b-4f8e-9f4c-13eec5040a5e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68280) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 596.786761] env[68280]: DEBUG oslo_concurrency.lockutils [None req-f56f28aa-2a0b-4f8e-9f4c-13eec5040a5e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68280) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 596.786911] env[68280]: DEBUG oslo_concurrency.lockutils [None req-f56f28aa-2a0b-4f8e-9f4c-13eec5040a5e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68280) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 596.787156] env[68280]: DEBUG oslo_concurrency.lockutils [None req-f56f28aa-2a0b-4f8e-9f4c-13eec5040a5e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68280) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 596.801874] env[68280]: DEBUG oslo_concurrency.lockutils [None req-f56f28aa-2a0b-4f8e-9f4c-13eec5040a5e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68280) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 596.802126] env[68280]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f56f28aa-2a0b-4f8e-9f4c-13eec5040a5e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 596.802263] env[68280]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f56f28aa-2a0b-4f8e-9f4c-13eec5040a5e None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=68280) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 598.804946] env[68280]: CRITICAL neutron [None req-f56f28aa-2a0b-4f8e-9f4c-13eec5040a5e None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 598.804946] env[68280]: ERROR neutron Traceback (most recent call last): [ 598.804946] env[68280]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 598.804946] env[68280]: ERROR neutron sys.exit(main()) [ 598.804946] env[68280]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 598.804946] env[68280]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 598.804946] env[68280]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 598.804946] env[68280]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 598.804946] env[68280]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 598.804946] env[68280]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 598.804946] env[68280]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 598.804946] env[68280]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 598.804946] env[68280]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 598.804946] env[68280]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 598.804946] env[68280]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 598.804946] env[68280]: ERROR neutron response = self.get(path=path, params=params) [ 598.805622] env[68280]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 598.805622] env[68280]: ERROR neutron raise RuntimeError(msg, last_err) [ 598.805622] env[68280]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 598.805622] env[68280]: ERROR neutron [ 601.435781] env[68493]: INFO neutron.common.config [-] Logging enabled! [ 601.436162] env[68493]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 601.436319] env[68493]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=68493) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 601.436545] env[68493]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 601.436634] env[68493]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 601.437492] env[68493]: WARNING oslo_policy.policy [None req-ca7448b6-e22b-4946-8e7d-be58d568c571 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 601.437710] env[68493]: WARNING oslo_policy.policy [None req-ca7448b6-e22b-4946-8e7d-be58d568c571 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 601.438926] env[68493]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-ca7448b6-e22b-4946-8e7d-be58d568c571 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 601.454183] env[68493]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ca7448b6-e22b-4946-8e7d-be58d568c571 None None] Session token - acquiring [ 601.455239] env[68493]: DEBUG networking_nsxv3.common.locking [None req-ca7448b6-e22b-4946-8e7d-be58d568c571 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68493) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 601.455450] env[68493]: DEBUG oslo_concurrency.lockutils [None req-ca7448b6-e22b-4946-8e7d-be58d568c571 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68493) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 601.455579] env[68493]: DEBUG oslo_concurrency.lockutils [None req-ca7448b6-e22b-4946-8e7d-be58d568c571 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68493) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 601.455988] env[68493]: DEBUG oslo_concurrency.lockutils [None req-ca7448b6-e22b-4946-8e7d-be58d568c571 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68493) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 601.466804] env[68493]: DEBUG oslo_concurrency.lockutils [None req-ca7448b6-e22b-4946-8e7d-be58d568c571 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68493) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 601.467104] env[68493]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ca7448b6-e22b-4946-8e7d-be58d568c571 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 601.467269] env[68493]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ca7448b6-e22b-4946-8e7d-be58d568c571 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=68493) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 603.476736] env[68493]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ca7448b6-e22b-4946-8e7d-be58d568c571 None None] Session token - acquiring [ 603.477069] env[68493]: DEBUG networking_nsxv3.common.locking [None req-ca7448b6-e22b-4946-8e7d-be58d568c571 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68493) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 603.477260] env[68493]: DEBUG oslo_concurrency.lockutils [None req-ca7448b6-e22b-4946-8e7d-be58d568c571 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68493) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 603.477383] env[68493]: DEBUG oslo_concurrency.lockutils [None req-ca7448b6-e22b-4946-8e7d-be58d568c571 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68493) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 603.477683] env[68493]: DEBUG oslo_concurrency.lockutils [None req-ca7448b6-e22b-4946-8e7d-be58d568c571 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68493) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 603.486964] env[68493]: DEBUG oslo_concurrency.lockutils [None req-ca7448b6-e22b-4946-8e7d-be58d568c571 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68493) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 603.487214] env[68493]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ca7448b6-e22b-4946-8e7d-be58d568c571 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 603.487593] env[68493]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ca7448b6-e22b-4946-8e7d-be58d568c571 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=68493) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 605.488894] env[68493]: CRITICAL neutron [None req-ca7448b6-e22b-4946-8e7d-be58d568c571 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 605.488894] env[68493]: ERROR neutron Traceback (most recent call last): [ 605.488894] env[68493]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 605.488894] env[68493]: ERROR neutron sys.exit(main()) [ 605.488894] env[68493]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 605.488894] env[68493]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 605.488894] env[68493]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 605.488894] env[68493]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 605.488894] env[68493]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 605.488894] env[68493]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 605.488894] env[68493]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 605.488894] env[68493]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 605.488894] env[68493]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 605.488894] env[68493]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 605.488894] env[68493]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 605.488894] env[68493]: ERROR neutron response = self.get(path=path, params=params) [ 605.489640] env[68493]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 605.489640] env[68493]: ERROR neutron raise RuntimeError(msg, last_err) [ 605.489640] env[68493]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 605.489640] env[68493]: ERROR neutron [ 608.208744] env[68736]: INFO neutron.common.config [-] Logging enabled! [ 608.209018] env[68736]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 608.210233] env[68736]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=68736) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 608.210233] env[68736]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 608.210233] env[68736]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 608.211355] env[68736]: WARNING oslo_policy.policy [None req-e2ae3325-a8f7-4277-ade6-0ec797e2019c None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 608.211722] env[68736]: WARNING oslo_policy.policy [None req-e2ae3325-a8f7-4277-ade6-0ec797e2019c None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 608.213656] env[68736]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-e2ae3325-a8f7-4277-ade6-0ec797e2019c None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 608.233494] env[68736]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e2ae3325-a8f7-4277-ade6-0ec797e2019c None None] Session token - acquiring [ 608.234860] env[68736]: DEBUG networking_nsxv3.common.locking [None req-e2ae3325-a8f7-4277-ade6-0ec797e2019c None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68736) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 608.235198] env[68736]: DEBUG oslo_concurrency.lockutils [None req-e2ae3325-a8f7-4277-ade6-0ec797e2019c None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68736) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 608.235406] env[68736]: DEBUG oslo_concurrency.lockutils [None req-e2ae3325-a8f7-4277-ade6-0ec797e2019c None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68736) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 608.236070] env[68736]: DEBUG oslo_concurrency.lockutils [None req-e2ae3325-a8f7-4277-ade6-0ec797e2019c None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68736) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 608.248411] env[68736]: DEBUG oslo_concurrency.lockutils [None req-e2ae3325-a8f7-4277-ade6-0ec797e2019c None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68736) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 608.248754] env[68736]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e2ae3325-a8f7-4277-ade6-0ec797e2019c None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 608.249022] env[68736]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e2ae3325-a8f7-4277-ade6-0ec797e2019c None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=68736) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 610.255309] env[68736]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e2ae3325-a8f7-4277-ade6-0ec797e2019c None None] Session token - acquiring [ 610.255649] env[68736]: DEBUG networking_nsxv3.common.locking [None req-e2ae3325-a8f7-4277-ade6-0ec797e2019c None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68736) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 610.255795] env[68736]: DEBUG oslo_concurrency.lockutils [None req-e2ae3325-a8f7-4277-ade6-0ec797e2019c None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68736) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 610.255923] env[68736]: DEBUG oslo_concurrency.lockutils [None req-e2ae3325-a8f7-4277-ade6-0ec797e2019c None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68736) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 610.256268] env[68736]: DEBUG oslo_concurrency.lockutils [None req-e2ae3325-a8f7-4277-ade6-0ec797e2019c None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68736) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 610.269352] env[68736]: DEBUG oslo_concurrency.lockutils [None req-e2ae3325-a8f7-4277-ade6-0ec797e2019c None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68736) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 610.269602] env[68736]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e2ae3325-a8f7-4277-ade6-0ec797e2019c None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 610.269736] env[68736]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e2ae3325-a8f7-4277-ade6-0ec797e2019c None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=68736) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 612.272669] env[68736]: CRITICAL neutron [None req-e2ae3325-a8f7-4277-ade6-0ec797e2019c None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 612.272669] env[68736]: ERROR neutron Traceback (most recent call last): [ 612.272669] env[68736]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 612.272669] env[68736]: ERROR neutron sys.exit(main()) [ 612.272669] env[68736]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 612.272669] env[68736]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 612.272669] env[68736]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 612.272669] env[68736]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 612.272669] env[68736]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 612.272669] env[68736]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 612.272669] env[68736]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 612.272669] env[68736]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 612.272669] env[68736]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 612.272669] env[68736]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 612.272669] env[68736]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 612.272669] env[68736]: ERROR neutron response = self.get(path=path, params=params) [ 612.273376] env[68736]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 612.273376] env[68736]: ERROR neutron raise RuntimeError(msg, last_err) [ 612.273376] env[68736]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 612.273376] env[68736]: ERROR neutron [ 615.046999] env[69085]: INFO neutron.common.config [-] Logging enabled! [ 615.047293] env[69085]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 615.047471] env[69085]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=69085) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 615.047711] env[69085]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 615.047809] env[69085]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 615.048692] env[69085]: WARNING oslo_policy.policy [None req-0702aced-7dc3-4f58-9567-05db7dc3f63e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 615.048911] env[69085]: WARNING oslo_policy.policy [None req-0702aced-7dc3-4f58-9567-05db7dc3f63e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 615.050077] env[69085]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-0702aced-7dc3-4f58-9567-05db7dc3f63e None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 615.071667] env[69085]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0702aced-7dc3-4f58-9567-05db7dc3f63e None None] Session token - acquiring [ 615.072730] env[69085]: DEBUG networking_nsxv3.common.locking [None req-0702aced-7dc3-4f58-9567-05db7dc3f63e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69085) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 615.072935] env[69085]: DEBUG oslo_concurrency.lockutils [None req-0702aced-7dc3-4f58-9567-05db7dc3f63e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69085) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 615.073059] env[69085]: DEBUG oslo_concurrency.lockutils [None req-0702aced-7dc3-4f58-9567-05db7dc3f63e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69085) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 615.073675] env[69085]: DEBUG oslo_concurrency.lockutils [None req-0702aced-7dc3-4f58-9567-05db7dc3f63e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69085) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 615.083601] env[69085]: DEBUG oslo_concurrency.lockutils [None req-0702aced-7dc3-4f58-9567-05db7dc3f63e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69085) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 615.083883] env[69085]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0702aced-7dc3-4f58-9567-05db7dc3f63e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 615.084041] env[69085]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0702aced-7dc3-4f58-9567-05db7dc3f63e None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=69085) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 617.090071] env[69085]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0702aced-7dc3-4f58-9567-05db7dc3f63e None None] Session token - acquiring [ 617.090544] env[69085]: DEBUG networking_nsxv3.common.locking [None req-0702aced-7dc3-4f58-9567-05db7dc3f63e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69085) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 617.090778] env[69085]: DEBUG oslo_concurrency.lockutils [None req-0702aced-7dc3-4f58-9567-05db7dc3f63e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69085) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 617.090910] env[69085]: DEBUG oslo_concurrency.lockutils [None req-0702aced-7dc3-4f58-9567-05db7dc3f63e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69085) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 617.091251] env[69085]: DEBUG oslo_concurrency.lockutils [None req-0702aced-7dc3-4f58-9567-05db7dc3f63e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69085) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 617.101048] env[69085]: DEBUG oslo_concurrency.lockutils [None req-0702aced-7dc3-4f58-9567-05db7dc3f63e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69085) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 617.102222] env[69085]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0702aced-7dc3-4f58-9567-05db7dc3f63e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 617.102222] env[69085]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0702aced-7dc3-4f58-9567-05db7dc3f63e None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=69085) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 619.105003] env[69085]: CRITICAL neutron [None req-0702aced-7dc3-4f58-9567-05db7dc3f63e None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 619.105003] env[69085]: ERROR neutron Traceback (most recent call last): [ 619.105003] env[69085]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 619.105003] env[69085]: ERROR neutron sys.exit(main()) [ 619.105003] env[69085]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 619.105003] env[69085]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 619.105003] env[69085]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 619.105003] env[69085]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 619.105003] env[69085]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 619.105003] env[69085]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 619.105003] env[69085]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 619.105003] env[69085]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 619.105003] env[69085]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 619.105003] env[69085]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 619.105003] env[69085]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 619.105003] env[69085]: ERROR neutron response = self.get(path=path, params=params) [ 619.105754] env[69085]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 619.105754] env[69085]: ERROR neutron raise RuntimeError(msg, last_err) [ 619.105754] env[69085]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 619.105754] env[69085]: ERROR neutron [ 621.827608] env[69245]: INFO neutron.common.config [-] Logging enabled! [ 621.827883] env[69245]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 621.828173] env[69245]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=69245) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 621.828398] env[69245]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 621.828492] env[69245]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 621.829333] env[69245]: WARNING oslo_policy.policy [None req-faf3a19b-c071-42b7-b163-8b2c72381c88 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 621.829552] env[69245]: WARNING oslo_policy.policy [None req-faf3a19b-c071-42b7-b163-8b2c72381c88 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 621.830762] env[69245]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-faf3a19b-c071-42b7-b163-8b2c72381c88 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 621.843719] env[69245]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-faf3a19b-c071-42b7-b163-8b2c72381c88 None None] Session token - acquiring [ 621.844467] env[69245]: DEBUG networking_nsxv3.common.locking [None req-faf3a19b-c071-42b7-b163-8b2c72381c88 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69245) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 621.844689] env[69245]: DEBUG oslo_concurrency.lockutils [None req-faf3a19b-c071-42b7-b163-8b2c72381c88 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69245) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 621.844890] env[69245]: DEBUG oslo_concurrency.lockutils [None req-faf3a19b-c071-42b7-b163-8b2c72381c88 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69245) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 621.845205] env[69245]: DEBUG oslo_concurrency.lockutils [None req-faf3a19b-c071-42b7-b163-8b2c72381c88 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69245) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 621.854723] env[69245]: DEBUG oslo_concurrency.lockutils [None req-faf3a19b-c071-42b7-b163-8b2c72381c88 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69245) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 621.854978] env[69245]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-faf3a19b-c071-42b7-b163-8b2c72381c88 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 621.855126] env[69245]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-faf3a19b-c071-42b7-b163-8b2c72381c88 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=69245) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 623.861197] env[69245]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-faf3a19b-c071-42b7-b163-8b2c72381c88 None None] Session token - acquiring [ 623.861654] env[69245]: DEBUG networking_nsxv3.common.locking [None req-faf3a19b-c071-42b7-b163-8b2c72381c88 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69245) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 623.861886] env[69245]: DEBUG oslo_concurrency.lockutils [None req-faf3a19b-c071-42b7-b163-8b2c72381c88 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69245) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 623.862032] env[69245]: DEBUG oslo_concurrency.lockutils [None req-faf3a19b-c071-42b7-b163-8b2c72381c88 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69245) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 623.862390] env[69245]: DEBUG oslo_concurrency.lockutils [None req-faf3a19b-c071-42b7-b163-8b2c72381c88 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69245) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 623.871740] env[69245]: DEBUG oslo_concurrency.lockutils [None req-faf3a19b-c071-42b7-b163-8b2c72381c88 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69245) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 623.871981] env[69245]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-faf3a19b-c071-42b7-b163-8b2c72381c88 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 623.872143] env[69245]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-faf3a19b-c071-42b7-b163-8b2c72381c88 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=69245) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 625.876950] env[69245]: CRITICAL neutron [None req-faf3a19b-c071-42b7-b163-8b2c72381c88 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 625.876950] env[69245]: ERROR neutron Traceback (most recent call last): [ 625.876950] env[69245]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 625.876950] env[69245]: ERROR neutron sys.exit(main()) [ 625.876950] env[69245]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 625.876950] env[69245]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 625.876950] env[69245]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 625.876950] env[69245]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 625.876950] env[69245]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 625.876950] env[69245]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 625.876950] env[69245]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 625.876950] env[69245]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 625.876950] env[69245]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 625.876950] env[69245]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 625.876950] env[69245]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 625.876950] env[69245]: ERROR neutron response = self.get(path=path, params=params) [ 625.877863] env[69245]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 625.877863] env[69245]: ERROR neutron raise RuntimeError(msg, last_err) [ 625.877863] env[69245]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 625.877863] env[69245]: ERROR neutron [ 628.630624] env[69459]: INFO neutron.common.config [-] Logging enabled! [ 628.630962] env[69459]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 628.631142] env[69459]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=69459) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 628.631375] env[69459]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 628.631468] env[69459]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 628.632320] env[69459]: WARNING oslo_policy.policy [None req-7036c6da-cb30-4354-b208-3159e792cd5a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 628.632541] env[69459]: WARNING oslo_policy.policy [None req-7036c6da-cb30-4354-b208-3159e792cd5a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 628.633746] env[69459]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-7036c6da-cb30-4354-b208-3159e792cd5a None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 628.649149] env[69459]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7036c6da-cb30-4354-b208-3159e792cd5a None None] Session token - acquiring [ 628.649232] env[69459]: DEBUG networking_nsxv3.common.locking [None req-7036c6da-cb30-4354-b208-3159e792cd5a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69459) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 628.649442] env[69459]: DEBUG oslo_concurrency.lockutils [None req-7036c6da-cb30-4354-b208-3159e792cd5a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69459) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 628.649569] env[69459]: DEBUG oslo_concurrency.lockutils [None req-7036c6da-cb30-4354-b208-3159e792cd5a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69459) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 628.649928] env[69459]: DEBUG oslo_concurrency.lockutils [None req-7036c6da-cb30-4354-b208-3159e792cd5a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69459) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 628.663346] env[69459]: DEBUG oslo_concurrency.lockutils [None req-7036c6da-cb30-4354-b208-3159e792cd5a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69459) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 628.663533] env[69459]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7036c6da-cb30-4354-b208-3159e792cd5a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 628.664121] env[69459]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7036c6da-cb30-4354-b208-3159e792cd5a None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=69459) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 630.669975] env[69459]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7036c6da-cb30-4354-b208-3159e792cd5a None None] Session token - acquiring [ 630.670383] env[69459]: DEBUG networking_nsxv3.common.locking [None req-7036c6da-cb30-4354-b208-3159e792cd5a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69459) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 630.670578] env[69459]: DEBUG oslo_concurrency.lockutils [None req-7036c6da-cb30-4354-b208-3159e792cd5a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69459) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 630.670702] env[69459]: DEBUG oslo_concurrency.lockutils [None req-7036c6da-cb30-4354-b208-3159e792cd5a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69459) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 630.670977] env[69459]: DEBUG oslo_concurrency.lockutils [None req-7036c6da-cb30-4354-b208-3159e792cd5a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69459) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 630.680093] env[69459]: DEBUG oslo_concurrency.lockutils [None req-7036c6da-cb30-4354-b208-3159e792cd5a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69459) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 630.680374] env[69459]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7036c6da-cb30-4354-b208-3159e792cd5a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 630.680468] env[69459]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7036c6da-cb30-4354-b208-3159e792cd5a None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=69459) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 632.683375] env[69459]: CRITICAL neutron [None req-7036c6da-cb30-4354-b208-3159e792cd5a None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 632.683375] env[69459]: ERROR neutron Traceback (most recent call last): [ 632.683375] env[69459]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 632.683375] env[69459]: ERROR neutron sys.exit(main()) [ 632.683375] env[69459]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 632.683375] env[69459]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 632.683375] env[69459]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 632.683375] env[69459]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 632.683375] env[69459]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 632.683375] env[69459]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 632.683375] env[69459]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 632.683375] env[69459]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 632.683375] env[69459]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 632.683375] env[69459]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 632.683375] env[69459]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 632.683375] env[69459]: ERROR neutron response = self.get(path=path, params=params) [ 632.684124] env[69459]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 632.684124] env[69459]: ERROR neutron raise RuntimeError(msg, last_err) [ 632.684124] env[69459]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 632.684124] env[69459]: ERROR neutron [ 635.076599] env[69527]: INFO neutron.common.config [-] Logging enabled! [ 635.076894] env[69527]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 635.077065] env[69527]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=69527) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 635.077304] env[69527]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 635.077395] env[69527]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 635.078209] env[69527]: WARNING oslo_policy.policy [None req-d4e32062-77b7-46cb-aca7-00a7b4d10d74 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 635.078427] env[69527]: WARNING oslo_policy.policy [None req-d4e32062-77b7-46cb-aca7-00a7b4d10d74 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 635.079607] env[69527]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-d4e32062-77b7-46cb-aca7-00a7b4d10d74 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 635.093755] env[69527]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d4e32062-77b7-46cb-aca7-00a7b4d10d74 None None] Session token - acquiring [ 635.094492] env[69527]: DEBUG networking_nsxv3.common.locking [None req-d4e32062-77b7-46cb-aca7-00a7b4d10d74 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69527) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 635.094704] env[69527]: DEBUG oslo_concurrency.lockutils [None req-d4e32062-77b7-46cb-aca7-00a7b4d10d74 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69527) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 635.094846] env[69527]: DEBUG oslo_concurrency.lockutils [None req-d4e32062-77b7-46cb-aca7-00a7b4d10d74 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69527) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 635.095206] env[69527]: DEBUG oslo_concurrency.lockutils [None req-d4e32062-77b7-46cb-aca7-00a7b4d10d74 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69527) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 635.104311] env[69527]: DEBUG oslo_concurrency.lockutils [None req-d4e32062-77b7-46cb-aca7-00a7b4d10d74 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69527) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 635.104554] env[69527]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d4e32062-77b7-46cb-aca7-00a7b4d10d74 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 635.104717] env[69527]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d4e32062-77b7-46cb-aca7-00a7b4d10d74 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=69527) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 637.111088] env[69527]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d4e32062-77b7-46cb-aca7-00a7b4d10d74 None None] Session token - acquiring [ 637.111616] env[69527]: DEBUG networking_nsxv3.common.locking [None req-d4e32062-77b7-46cb-aca7-00a7b4d10d74 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69527) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 637.111926] env[69527]: DEBUG oslo_concurrency.lockutils [None req-d4e32062-77b7-46cb-aca7-00a7b4d10d74 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69527) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 637.112162] env[69527]: DEBUG oslo_concurrency.lockutils [None req-d4e32062-77b7-46cb-aca7-00a7b4d10d74 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69527) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 637.112526] env[69527]: DEBUG oslo_concurrency.lockutils [None req-d4e32062-77b7-46cb-aca7-00a7b4d10d74 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69527) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 637.121873] env[69527]: DEBUG oslo_concurrency.lockutils [None req-d4e32062-77b7-46cb-aca7-00a7b4d10d74 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69527) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 637.122122] env[69527]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d4e32062-77b7-46cb-aca7-00a7b4d10d74 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 637.122266] env[69527]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d4e32062-77b7-46cb-aca7-00a7b4d10d74 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=69527) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 639.124919] env[69527]: CRITICAL neutron [None req-d4e32062-77b7-46cb-aca7-00a7b4d10d74 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 639.124919] env[69527]: ERROR neutron Traceback (most recent call last): [ 639.124919] env[69527]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 639.124919] env[69527]: ERROR neutron sys.exit(main()) [ 639.124919] env[69527]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 639.124919] env[69527]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 639.124919] env[69527]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 639.124919] env[69527]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 639.124919] env[69527]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 639.124919] env[69527]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 639.124919] env[69527]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 639.124919] env[69527]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 639.124919] env[69527]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 639.124919] env[69527]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 639.124919] env[69527]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 639.124919] env[69527]: ERROR neutron response = self.get(path=path, params=params) [ 639.125640] env[69527]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 639.125640] env[69527]: ERROR neutron raise RuntimeError(msg, last_err) [ 639.125640] env[69527]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 639.125640] env[69527]: ERROR neutron [ 641.631743] env[69592]: INFO neutron.common.config [-] Logging enabled! [ 641.632062] env[69592]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 641.632301] env[69592]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=69592) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 641.632560] env[69592]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 641.632659] env[69592]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 641.633572] env[69592]: WARNING oslo_policy.policy [None req-1960acb5-5ba3-4304-84a4-4466d9f30c6b None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 641.633807] env[69592]: WARNING oslo_policy.policy [None req-1960acb5-5ba3-4304-84a4-4466d9f30c6b None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 641.635065] env[69592]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-1960acb5-5ba3-4304-84a4-4466d9f30c6b None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 641.650554] env[69592]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1960acb5-5ba3-4304-84a4-4466d9f30c6b None None] Session token - acquiring [ 641.651377] env[69592]: DEBUG networking_nsxv3.common.locking [None req-1960acb5-5ba3-4304-84a4-4466d9f30c6b None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69592) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 641.651585] env[69592]: DEBUG oslo_concurrency.lockutils [None req-1960acb5-5ba3-4304-84a4-4466d9f30c6b None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69592) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 641.651727] env[69592]: DEBUG oslo_concurrency.lockutils [None req-1960acb5-5ba3-4304-84a4-4466d9f30c6b None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69592) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 641.652118] env[69592]: DEBUG oslo_concurrency.lockutils [None req-1960acb5-5ba3-4304-84a4-4466d9f30c6b None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69592) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 641.667624] env[69592]: DEBUG oslo_concurrency.lockutils [None req-1960acb5-5ba3-4304-84a4-4466d9f30c6b None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69592) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 641.667900] env[69592]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1960acb5-5ba3-4304-84a4-4466d9f30c6b None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 641.668064] env[69592]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1960acb5-5ba3-4304-84a4-4466d9f30c6b None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=69592) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 643.673892] env[69592]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1960acb5-5ba3-4304-84a4-4466d9f30c6b None None] Session token - acquiring [ 643.674334] env[69592]: DEBUG networking_nsxv3.common.locking [None req-1960acb5-5ba3-4304-84a4-4466d9f30c6b None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69592) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 643.674445] env[69592]: DEBUG oslo_concurrency.lockutils [None req-1960acb5-5ba3-4304-84a4-4466d9f30c6b None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69592) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 643.674567] env[69592]: DEBUG oslo_concurrency.lockutils [None req-1960acb5-5ba3-4304-84a4-4466d9f30c6b None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69592) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 643.674862] env[69592]: DEBUG oslo_concurrency.lockutils [None req-1960acb5-5ba3-4304-84a4-4466d9f30c6b None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69592) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 643.683894] env[69592]: DEBUG oslo_concurrency.lockutils [None req-1960acb5-5ba3-4304-84a4-4466d9f30c6b None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69592) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 643.684320] env[69592]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1960acb5-5ba3-4304-84a4-4466d9f30c6b None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 643.684400] env[69592]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1960acb5-5ba3-4304-84a4-4466d9f30c6b None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=69592) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 645.687306] env[69592]: CRITICAL neutron [None req-1960acb5-5ba3-4304-84a4-4466d9f30c6b None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 645.687306] env[69592]: ERROR neutron Traceback (most recent call last): [ 645.687306] env[69592]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 645.687306] env[69592]: ERROR neutron sys.exit(main()) [ 645.687306] env[69592]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 645.687306] env[69592]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 645.687306] env[69592]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 645.687306] env[69592]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 645.687306] env[69592]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 645.687306] env[69592]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 645.687306] env[69592]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 645.687306] env[69592]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 645.687306] env[69592]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 645.687306] env[69592]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 645.687306] env[69592]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 645.687306] env[69592]: ERROR neutron response = self.get(path=path, params=params) [ 645.687996] env[69592]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 645.687996] env[69592]: ERROR neutron raise RuntimeError(msg, last_err) [ 645.687996] env[69592]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 645.687996] env[69592]: ERROR neutron [ 648.399119] env[69669]: INFO neutron.common.config [-] Logging enabled! [ 648.399498] env[69669]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 648.399688] env[69669]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=69669) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 648.399925] env[69669]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 648.400035] env[69669]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 648.400877] env[69669]: WARNING oslo_policy.policy [None req-8da31e3e-01c0-40e1-aa33-acf45b5bfc63 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 648.401084] env[69669]: WARNING oslo_policy.policy [None req-8da31e3e-01c0-40e1-aa33-acf45b5bfc63 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 648.402300] env[69669]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-8da31e3e-01c0-40e1-aa33-acf45b5bfc63 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 648.418260] env[69669]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8da31e3e-01c0-40e1-aa33-acf45b5bfc63 None None] Session token - acquiring [ 648.419081] env[69669]: DEBUG networking_nsxv3.common.locking [None req-8da31e3e-01c0-40e1-aa33-acf45b5bfc63 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69669) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 648.419276] env[69669]: DEBUG oslo_concurrency.lockutils [None req-8da31e3e-01c0-40e1-aa33-acf45b5bfc63 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69669) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 648.419399] env[69669]: DEBUG oslo_concurrency.lockutils [None req-8da31e3e-01c0-40e1-aa33-acf45b5bfc63 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69669) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 648.420779] env[69669]: DEBUG oslo_concurrency.lockutils [None req-8da31e3e-01c0-40e1-aa33-acf45b5bfc63 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69669) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 648.428638] env[69669]: DEBUG oslo_concurrency.lockutils [None req-8da31e3e-01c0-40e1-aa33-acf45b5bfc63 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69669) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 648.428870] env[69669]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8da31e3e-01c0-40e1-aa33-acf45b5bfc63 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 648.429008] env[69669]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8da31e3e-01c0-40e1-aa33-acf45b5bfc63 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=69669) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 650.435777] env[69669]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8da31e3e-01c0-40e1-aa33-acf45b5bfc63 None None] Session token - acquiring [ 650.435777] env[69669]: DEBUG networking_nsxv3.common.locking [None req-8da31e3e-01c0-40e1-aa33-acf45b5bfc63 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69669) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 650.436258] env[69669]: DEBUG oslo_concurrency.lockutils [None req-8da31e3e-01c0-40e1-aa33-acf45b5bfc63 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69669) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 650.436258] env[69669]: DEBUG oslo_concurrency.lockutils [None req-8da31e3e-01c0-40e1-aa33-acf45b5bfc63 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69669) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 650.436395] env[69669]: DEBUG oslo_concurrency.lockutils [None req-8da31e3e-01c0-40e1-aa33-acf45b5bfc63 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69669) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 650.457099] env[69669]: DEBUG oslo_concurrency.lockutils [None req-8da31e3e-01c0-40e1-aa33-acf45b5bfc63 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69669) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 650.457361] env[69669]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8da31e3e-01c0-40e1-aa33-acf45b5bfc63 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 650.457494] env[69669]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8da31e3e-01c0-40e1-aa33-acf45b5bfc63 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=69669) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 652.460425] env[69669]: CRITICAL neutron [None req-8da31e3e-01c0-40e1-aa33-acf45b5bfc63 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 652.460425] env[69669]: ERROR neutron Traceback (most recent call last): [ 652.460425] env[69669]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 652.460425] env[69669]: ERROR neutron sys.exit(main()) [ 652.460425] env[69669]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 652.460425] env[69669]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 652.460425] env[69669]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 652.460425] env[69669]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 652.460425] env[69669]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 652.460425] env[69669]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 652.460425] env[69669]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 652.460425] env[69669]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 652.460425] env[69669]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 652.460425] env[69669]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 652.460425] env[69669]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 652.460425] env[69669]: ERROR neutron response = self.get(path=path, params=params) [ 652.461101] env[69669]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 652.461101] env[69669]: ERROR neutron raise RuntimeError(msg, last_err) [ 652.461101] env[69669]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 652.461101] env[69669]: ERROR neutron [ 655.126651] env[69801]: INFO neutron.common.config [-] Logging enabled! [ 655.126971] env[69801]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 655.127312] env[69801]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=69801) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 655.127472] env[69801]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 655.127552] env[69801]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 655.128426] env[69801]: WARNING oslo_policy.policy [None req-73314f6e-bc0f-44e6-882e-4fd0c5600d4d None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 655.128665] env[69801]: WARNING oslo_policy.policy [None req-73314f6e-bc0f-44e6-882e-4fd0c5600d4d None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 655.129889] env[69801]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-73314f6e-bc0f-44e6-882e-4fd0c5600d4d None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 655.144288] env[69801]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-73314f6e-bc0f-44e6-882e-4fd0c5600d4d None None] Session token - acquiring [ 655.145204] env[69801]: DEBUG networking_nsxv3.common.locking [None req-73314f6e-bc0f-44e6-882e-4fd0c5600d4d None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69801) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 655.145446] env[69801]: DEBUG oslo_concurrency.lockutils [None req-73314f6e-bc0f-44e6-882e-4fd0c5600d4d None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69801) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 655.145539] env[69801]: DEBUG oslo_concurrency.lockutils [None req-73314f6e-bc0f-44e6-882e-4fd0c5600d4d None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69801) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 655.145903] env[69801]: DEBUG oslo_concurrency.lockutils [None req-73314f6e-bc0f-44e6-882e-4fd0c5600d4d None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69801) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 655.155928] env[69801]: DEBUG oslo_concurrency.lockutils [None req-73314f6e-bc0f-44e6-882e-4fd0c5600d4d None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69801) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 655.156248] env[69801]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-73314f6e-bc0f-44e6-882e-4fd0c5600d4d None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 655.156395] env[69801]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-73314f6e-bc0f-44e6-882e-4fd0c5600d4d None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=69801) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 657.164402] env[69801]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-73314f6e-bc0f-44e6-882e-4fd0c5600d4d None None] Session token - acquiring [ 657.164811] env[69801]: DEBUG networking_nsxv3.common.locking [None req-73314f6e-bc0f-44e6-882e-4fd0c5600d4d None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69801) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 657.165030] env[69801]: DEBUG oslo_concurrency.lockutils [None req-73314f6e-bc0f-44e6-882e-4fd0c5600d4d None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69801) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 657.165123] env[69801]: DEBUG oslo_concurrency.lockutils [None req-73314f6e-bc0f-44e6-882e-4fd0c5600d4d None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69801) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 657.165391] env[69801]: DEBUG oslo_concurrency.lockutils [None req-73314f6e-bc0f-44e6-882e-4fd0c5600d4d None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69801) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 657.174677] env[69801]: DEBUG oslo_concurrency.lockutils [None req-73314f6e-bc0f-44e6-882e-4fd0c5600d4d None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69801) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 657.174978] env[69801]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-73314f6e-bc0f-44e6-882e-4fd0c5600d4d None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 657.175116] env[69801]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-73314f6e-bc0f-44e6-882e-4fd0c5600d4d None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=69801) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 659.176898] env[69801]: CRITICAL neutron [None req-73314f6e-bc0f-44e6-882e-4fd0c5600d4d None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 659.176898] env[69801]: ERROR neutron Traceback (most recent call last): [ 659.176898] env[69801]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 659.176898] env[69801]: ERROR neutron sys.exit(main()) [ 659.176898] env[69801]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 659.176898] env[69801]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 659.176898] env[69801]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 659.176898] env[69801]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 659.176898] env[69801]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 659.176898] env[69801]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 659.176898] env[69801]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 659.176898] env[69801]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 659.176898] env[69801]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 659.176898] env[69801]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 659.176898] env[69801]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 659.176898] env[69801]: ERROR neutron response = self.get(path=path, params=params) [ 659.177686] env[69801]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 659.177686] env[69801]: ERROR neutron raise RuntimeError(msg, last_err) [ 659.177686] env[69801]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 659.177686] env[69801]: ERROR neutron [ 661.949769] env[69989]: INFO neutron.common.config [-] Logging enabled! [ 661.950175] env[69989]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 661.950313] env[69989]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=69989) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 661.950554] env[69989]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 661.950650] env[69989]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 661.951507] env[69989]: WARNING oslo_policy.policy [None req-0515df75-72ce-445e-9af2-5fb8a4d17962 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 661.951723] env[69989]: WARNING oslo_policy.policy [None req-0515df75-72ce-445e-9af2-5fb8a4d17962 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 661.953304] env[69989]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-0515df75-72ce-445e-9af2-5fb8a4d17962 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 661.969535] env[69989]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0515df75-72ce-445e-9af2-5fb8a4d17962 None None] Session token - acquiring [ 661.970469] env[69989]: DEBUG networking_nsxv3.common.locking [None req-0515df75-72ce-445e-9af2-5fb8a4d17962 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69989) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 661.970653] env[69989]: DEBUG oslo_concurrency.lockutils [None req-0515df75-72ce-445e-9af2-5fb8a4d17962 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69989) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 661.970785] env[69989]: DEBUG oslo_concurrency.lockutils [None req-0515df75-72ce-445e-9af2-5fb8a4d17962 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69989) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 661.971160] env[69989]: DEBUG oslo_concurrency.lockutils [None req-0515df75-72ce-445e-9af2-5fb8a4d17962 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69989) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 661.982377] env[69989]: DEBUG oslo_concurrency.lockutils [None req-0515df75-72ce-445e-9af2-5fb8a4d17962 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69989) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 661.982555] env[69989]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0515df75-72ce-445e-9af2-5fb8a4d17962 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 661.982706] env[69989]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0515df75-72ce-445e-9af2-5fb8a4d17962 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=69989) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 663.989011] env[69989]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0515df75-72ce-445e-9af2-5fb8a4d17962 None None] Session token - acquiring [ 663.989333] env[69989]: DEBUG networking_nsxv3.common.locking [None req-0515df75-72ce-445e-9af2-5fb8a4d17962 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69989) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 663.989513] env[69989]: DEBUG oslo_concurrency.lockutils [None req-0515df75-72ce-445e-9af2-5fb8a4d17962 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69989) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 663.989691] env[69989]: DEBUG oslo_concurrency.lockutils [None req-0515df75-72ce-445e-9af2-5fb8a4d17962 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69989) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 663.990029] env[69989]: DEBUG oslo_concurrency.lockutils [None req-0515df75-72ce-445e-9af2-5fb8a4d17962 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69989) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 663.999678] env[69989]: DEBUG oslo_concurrency.lockutils [None req-0515df75-72ce-445e-9af2-5fb8a4d17962 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69989) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 663.999678] env[69989]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0515df75-72ce-445e-9af2-5fb8a4d17962 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 663.999806] env[69989]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0515df75-72ce-445e-9af2-5fb8a4d17962 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=69989) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 666.001026] env[69989]: CRITICAL neutron [None req-0515df75-72ce-445e-9af2-5fb8a4d17962 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 666.001026] env[69989]: ERROR neutron Traceback (most recent call last): [ 666.001026] env[69989]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 666.001026] env[69989]: ERROR neutron sys.exit(main()) [ 666.001026] env[69989]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 666.001026] env[69989]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 666.001026] env[69989]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 666.001026] env[69989]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 666.001026] env[69989]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 666.001026] env[69989]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 666.001026] env[69989]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 666.001026] env[69989]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 666.001026] env[69989]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 666.001026] env[69989]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 666.001026] env[69989]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 666.001026] env[69989]: ERROR neutron response = self.get(path=path, params=params) [ 666.001965] env[69989]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 666.001965] env[69989]: ERROR neutron raise RuntimeError(msg, last_err) [ 666.001965] env[69989]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 666.001965] env[69989]: ERROR neutron [ 668.763203] env[70124]: INFO neutron.common.config [-] Logging enabled! [ 668.763525] env[70124]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 668.763776] env[70124]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=70124) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 668.764016] env[70124]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 668.764115] env[70124]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 668.764992] env[70124]: WARNING oslo_policy.policy [None req-6ffa9081-0dd7-4a12-9eca-a93236defe77 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 668.765218] env[70124]: WARNING oslo_policy.policy [None req-6ffa9081-0dd7-4a12-9eca-a93236defe77 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 668.766449] env[70124]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-6ffa9081-0dd7-4a12-9eca-a93236defe77 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 668.781003] env[70124]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6ffa9081-0dd7-4a12-9eca-a93236defe77 None None] Session token - acquiring [ 668.781872] env[70124]: DEBUG networking_nsxv3.common.locking [None req-6ffa9081-0dd7-4a12-9eca-a93236defe77 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70124) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 668.782084] env[70124]: DEBUG oslo_concurrency.lockutils [None req-6ffa9081-0dd7-4a12-9eca-a93236defe77 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70124) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 668.782208] env[70124]: DEBUG oslo_concurrency.lockutils [None req-6ffa9081-0dd7-4a12-9eca-a93236defe77 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70124) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 668.782614] env[70124]: DEBUG oslo_concurrency.lockutils [None req-6ffa9081-0dd7-4a12-9eca-a93236defe77 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70124) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 668.792578] env[70124]: DEBUG oslo_concurrency.lockutils [None req-6ffa9081-0dd7-4a12-9eca-a93236defe77 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70124) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 668.792848] env[70124]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6ffa9081-0dd7-4a12-9eca-a93236defe77 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 668.792997] env[70124]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6ffa9081-0dd7-4a12-9eca-a93236defe77 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=70124) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 670.799059] env[70124]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6ffa9081-0dd7-4a12-9eca-a93236defe77 None None] Session token - acquiring [ 670.799376] env[70124]: DEBUG networking_nsxv3.common.locking [None req-6ffa9081-0dd7-4a12-9eca-a93236defe77 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70124) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 670.799569] env[70124]: DEBUG oslo_concurrency.lockutils [None req-6ffa9081-0dd7-4a12-9eca-a93236defe77 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70124) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 670.799694] env[70124]: DEBUG oslo_concurrency.lockutils [None req-6ffa9081-0dd7-4a12-9eca-a93236defe77 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70124) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 670.800033] env[70124]: DEBUG oslo_concurrency.lockutils [None req-6ffa9081-0dd7-4a12-9eca-a93236defe77 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70124) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 670.809366] env[70124]: DEBUG oslo_concurrency.lockutils [None req-6ffa9081-0dd7-4a12-9eca-a93236defe77 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70124) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 670.809611] env[70124]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6ffa9081-0dd7-4a12-9eca-a93236defe77 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 670.809751] env[70124]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6ffa9081-0dd7-4a12-9eca-a93236defe77 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=70124) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 672.812715] env[70124]: CRITICAL neutron [None req-6ffa9081-0dd7-4a12-9eca-a93236defe77 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 672.812715] env[70124]: ERROR neutron Traceback (most recent call last): [ 672.812715] env[70124]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 672.812715] env[70124]: ERROR neutron sys.exit(main()) [ 672.812715] env[70124]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 672.812715] env[70124]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 672.812715] env[70124]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 672.812715] env[70124]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 672.812715] env[70124]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 672.812715] env[70124]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 672.812715] env[70124]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 672.812715] env[70124]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 672.812715] env[70124]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 672.812715] env[70124]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 672.812715] env[70124]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 672.812715] env[70124]: ERROR neutron response = self.get(path=path, params=params) [ 672.813623] env[70124]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 672.813623] env[70124]: ERROR neutron raise RuntimeError(msg, last_err) [ 672.813623] env[70124]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 672.813623] env[70124]: ERROR neutron [ 675.384298] env[70306]: INFO neutron.common.config [-] Logging enabled! [ 675.384640] env[70306]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 675.384866] env[70306]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=70306) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 675.385097] env[70306]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 675.385191] env[70306]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 675.386009] env[70306]: WARNING oslo_policy.policy [None req-e798e1ce-bb2e-4844-86f9-06f20f084666 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 675.386233] env[70306]: WARNING oslo_policy.policy [None req-e798e1ce-bb2e-4844-86f9-06f20f084666 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 675.387421] env[70306]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-e798e1ce-bb2e-4844-86f9-06f20f084666 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 675.401645] env[70306]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e798e1ce-bb2e-4844-86f9-06f20f084666 None None] Session token - acquiring [ 675.402521] env[70306]: DEBUG networking_nsxv3.common.locking [None req-e798e1ce-bb2e-4844-86f9-06f20f084666 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70306) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 675.402748] env[70306]: DEBUG oslo_concurrency.lockutils [None req-e798e1ce-bb2e-4844-86f9-06f20f084666 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70306) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 675.402969] env[70306]: DEBUG oslo_concurrency.lockutils [None req-e798e1ce-bb2e-4844-86f9-06f20f084666 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70306) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 675.403441] env[70306]: DEBUG oslo_concurrency.lockutils [None req-e798e1ce-bb2e-4844-86f9-06f20f084666 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70306) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 675.413580] env[70306]: DEBUG oslo_concurrency.lockutils [None req-e798e1ce-bb2e-4844-86f9-06f20f084666 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70306) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 675.413892] env[70306]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e798e1ce-bb2e-4844-86f9-06f20f084666 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 675.413988] env[70306]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e798e1ce-bb2e-4844-86f9-06f20f084666 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=70306) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 677.420071] env[70306]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e798e1ce-bb2e-4844-86f9-06f20f084666 None None] Session token - acquiring [ 677.420388] env[70306]: DEBUG networking_nsxv3.common.locking [None req-e798e1ce-bb2e-4844-86f9-06f20f084666 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70306) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 677.420991] env[70306]: DEBUG oslo_concurrency.lockutils [None req-e798e1ce-bb2e-4844-86f9-06f20f084666 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70306) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 677.420991] env[70306]: DEBUG oslo_concurrency.lockutils [None req-e798e1ce-bb2e-4844-86f9-06f20f084666 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70306) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 677.420991] env[70306]: DEBUG oslo_concurrency.lockutils [None req-e798e1ce-bb2e-4844-86f9-06f20f084666 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70306) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 677.434513] env[70306]: DEBUG oslo_concurrency.lockutils [None req-e798e1ce-bb2e-4844-86f9-06f20f084666 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70306) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 677.434513] env[70306]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e798e1ce-bb2e-4844-86f9-06f20f084666 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 677.434513] env[70306]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e798e1ce-bb2e-4844-86f9-06f20f084666 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=70306) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 679.435458] env[70306]: CRITICAL neutron [None req-e798e1ce-bb2e-4844-86f9-06f20f084666 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 679.435458] env[70306]: ERROR neutron Traceback (most recent call last): [ 679.435458] env[70306]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 679.435458] env[70306]: ERROR neutron sys.exit(main()) [ 679.435458] env[70306]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 679.435458] env[70306]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 679.435458] env[70306]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 679.435458] env[70306]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 679.435458] env[70306]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 679.435458] env[70306]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 679.435458] env[70306]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 679.435458] env[70306]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 679.435458] env[70306]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 679.435458] env[70306]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 679.435458] env[70306]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 679.435458] env[70306]: ERROR neutron response = self.get(path=path, params=params) [ 679.436465] env[70306]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 679.436465] env[70306]: ERROR neutron raise RuntimeError(msg, last_err) [ 679.436465] env[70306]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 679.436465] env[70306]: ERROR neutron [ 682.300479] env[70408]: INFO neutron.common.config [-] Logging enabled! [ 682.300479] env[70408]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 682.300479] env[70408]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=70408) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 682.300479] env[70408]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 682.300479] env[70408]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 682.300479] env[70408]: WARNING oslo_policy.policy [None req-ffe5a861-243f-4659-afa0-c4b788521bcc None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 682.300479] env[70408]: WARNING oslo_policy.policy [None req-ffe5a861-243f-4659-afa0-c4b788521bcc None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 682.302454] env[70408]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-ffe5a861-243f-4659-afa0-c4b788521bcc None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 682.317733] env[70408]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ffe5a861-243f-4659-afa0-c4b788521bcc None None] Session token - acquiring [ 682.318578] env[70408]: DEBUG networking_nsxv3.common.locking [None req-ffe5a861-243f-4659-afa0-c4b788521bcc None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70408) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 682.318786] env[70408]: DEBUG oslo_concurrency.lockutils [None req-ffe5a861-243f-4659-afa0-c4b788521bcc None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70408) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 682.318915] env[70408]: DEBUG oslo_concurrency.lockutils [None req-ffe5a861-243f-4659-afa0-c4b788521bcc None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70408) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 682.319417] env[70408]: DEBUG oslo_concurrency.lockutils [None req-ffe5a861-243f-4659-afa0-c4b788521bcc None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70408) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 682.329366] env[70408]: DEBUG oslo_concurrency.lockutils [None req-ffe5a861-243f-4659-afa0-c4b788521bcc None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70408) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 682.329654] env[70408]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ffe5a861-243f-4659-afa0-c4b788521bcc None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 682.329808] env[70408]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ffe5a861-243f-4659-afa0-c4b788521bcc None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=70408) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 684.336746] env[70408]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ffe5a861-243f-4659-afa0-c4b788521bcc None None] Session token - acquiring [ 684.337066] env[70408]: DEBUG networking_nsxv3.common.locking [None req-ffe5a861-243f-4659-afa0-c4b788521bcc None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70408) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 684.337211] env[70408]: DEBUG oslo_concurrency.lockutils [None req-ffe5a861-243f-4659-afa0-c4b788521bcc None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70408) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 684.337325] env[70408]: DEBUG oslo_concurrency.lockutils [None req-ffe5a861-243f-4659-afa0-c4b788521bcc None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70408) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 684.337596] env[70408]: DEBUG oslo_concurrency.lockutils [None req-ffe5a861-243f-4659-afa0-c4b788521bcc None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70408) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 684.349213] env[70408]: DEBUG oslo_concurrency.lockutils [None req-ffe5a861-243f-4659-afa0-c4b788521bcc None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70408) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 684.349507] env[70408]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ffe5a861-243f-4659-afa0-c4b788521bcc None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 684.349645] env[70408]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ffe5a861-243f-4659-afa0-c4b788521bcc None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=70408) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 686.354140] env[70408]: CRITICAL neutron [None req-ffe5a861-243f-4659-afa0-c4b788521bcc None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 686.354140] env[70408]: ERROR neutron Traceback (most recent call last): [ 686.354140] env[70408]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 686.354140] env[70408]: ERROR neutron sys.exit(main()) [ 686.354140] env[70408]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 686.354140] env[70408]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 686.354140] env[70408]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 686.354140] env[70408]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 686.354140] env[70408]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 686.354140] env[70408]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 686.354140] env[70408]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 686.354140] env[70408]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 686.354140] env[70408]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 686.354140] env[70408]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 686.354140] env[70408]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 686.354140] env[70408]: ERROR neutron response = self.get(path=path, params=params) [ 686.354900] env[70408]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 686.354900] env[70408]: ERROR neutron raise RuntimeError(msg, last_err) [ 686.354900] env[70408]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 686.354900] env[70408]: ERROR neutron [ 689.327023] env[70639]: INFO neutron.common.config [-] Logging enabled! [ 689.327287] env[70639]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 689.327874] env[70639]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=70639) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 689.328198] env[70639]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 689.328299] env[70639]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 689.329525] env[70639]: WARNING oslo_policy.policy [None req-5863e889-8d49-4cc7-9ff9-95504f6a7dac None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 689.329750] env[70639]: WARNING oslo_policy.policy [None req-5863e889-8d49-4cc7-9ff9-95504f6a7dac None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 689.331017] env[70639]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-5863e889-8d49-4cc7-9ff9-95504f6a7dac None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 689.347999] env[70639]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5863e889-8d49-4cc7-9ff9-95504f6a7dac None None] Session token - acquiring [ 689.348890] env[70639]: DEBUG networking_nsxv3.common.locking [None req-5863e889-8d49-4cc7-9ff9-95504f6a7dac None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70639) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 689.349103] env[70639]: DEBUG oslo_concurrency.lockutils [None req-5863e889-8d49-4cc7-9ff9-95504f6a7dac None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70639) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 689.349466] env[70639]: DEBUG oslo_concurrency.lockutils [None req-5863e889-8d49-4cc7-9ff9-95504f6a7dac None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70639) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 689.349719] env[70639]: DEBUG oslo_concurrency.lockutils [None req-5863e889-8d49-4cc7-9ff9-95504f6a7dac None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70639) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 689.360699] env[70639]: DEBUG oslo_concurrency.lockutils [None req-5863e889-8d49-4cc7-9ff9-95504f6a7dac None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70639) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 689.361014] env[70639]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5863e889-8d49-4cc7-9ff9-95504f6a7dac None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 689.361165] env[70639]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5863e889-8d49-4cc7-9ff9-95504f6a7dac None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=70639) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 691.367278] env[70639]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5863e889-8d49-4cc7-9ff9-95504f6a7dac None None] Session token - acquiring [ 691.367690] env[70639]: DEBUG networking_nsxv3.common.locking [None req-5863e889-8d49-4cc7-9ff9-95504f6a7dac None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70639) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 691.367790] env[70639]: DEBUG oslo_concurrency.lockutils [None req-5863e889-8d49-4cc7-9ff9-95504f6a7dac None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70639) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 691.367945] env[70639]: DEBUG oslo_concurrency.lockutils [None req-5863e889-8d49-4cc7-9ff9-95504f6a7dac None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70639) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 691.368308] env[70639]: DEBUG oslo_concurrency.lockutils [None req-5863e889-8d49-4cc7-9ff9-95504f6a7dac None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70639) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 691.377990] env[70639]: DEBUG oslo_concurrency.lockutils [None req-5863e889-8d49-4cc7-9ff9-95504f6a7dac None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70639) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 691.378252] env[70639]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5863e889-8d49-4cc7-9ff9-95504f6a7dac None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 691.378394] env[70639]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5863e889-8d49-4cc7-9ff9-95504f6a7dac None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=70639) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 693.380965] env[70639]: CRITICAL neutron [None req-5863e889-8d49-4cc7-9ff9-95504f6a7dac None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 693.380965] env[70639]: ERROR neutron Traceback (most recent call last): [ 693.380965] env[70639]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 693.380965] env[70639]: ERROR neutron sys.exit(main()) [ 693.380965] env[70639]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 693.380965] env[70639]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 693.380965] env[70639]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 693.380965] env[70639]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 693.380965] env[70639]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 693.380965] env[70639]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 693.380965] env[70639]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 693.380965] env[70639]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 693.380965] env[70639]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 693.380965] env[70639]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 693.380965] env[70639]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 693.380965] env[70639]: ERROR neutron response = self.get(path=path, params=params) [ 693.381888] env[70639]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 693.381888] env[70639]: ERROR neutron raise RuntimeError(msg, last_err) [ 693.381888] env[70639]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 693.381888] env[70639]: ERROR neutron [ 696.063221] env[71011]: INFO neutron.common.config [-] Logging enabled! [ 696.063534] env[71011]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 696.064089] env[71011]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=71011) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 696.064335] env[71011]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 696.064461] env[71011]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 696.065293] env[71011]: WARNING oslo_policy.policy [None req-59bbe742-6904-44d5-82f1-ce8e482a3a79 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 696.065499] env[71011]: WARNING oslo_policy.policy [None req-59bbe742-6904-44d5-82f1-ce8e482a3a79 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 696.066704] env[71011]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-59bbe742-6904-44d5-82f1-ce8e482a3a79 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 696.085506] env[71011]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-59bbe742-6904-44d5-82f1-ce8e482a3a79 None None] Session token - acquiring [ 696.086453] env[71011]: DEBUG networking_nsxv3.common.locking [None req-59bbe742-6904-44d5-82f1-ce8e482a3a79 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71011) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 696.086806] env[71011]: DEBUG oslo_concurrency.lockutils [None req-59bbe742-6904-44d5-82f1-ce8e482a3a79 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71011) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 696.087331] env[71011]: DEBUG oslo_concurrency.lockutils [None req-59bbe742-6904-44d5-82f1-ce8e482a3a79 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71011) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 696.087823] env[71011]: DEBUG oslo_concurrency.lockutils [None req-59bbe742-6904-44d5-82f1-ce8e482a3a79 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71011) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 696.098084] env[71011]: DEBUG oslo_concurrency.lockutils [None req-59bbe742-6904-44d5-82f1-ce8e482a3a79 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71011) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 696.098529] env[71011]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-59bbe742-6904-44d5-82f1-ce8e482a3a79 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 696.098803] env[71011]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-59bbe742-6904-44d5-82f1-ce8e482a3a79 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=71011) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 698.109378] env[71011]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-59bbe742-6904-44d5-82f1-ce8e482a3a79 None None] Session token - acquiring [ 698.109633] env[71011]: DEBUG networking_nsxv3.common.locking [None req-59bbe742-6904-44d5-82f1-ce8e482a3a79 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71011) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 698.109633] env[71011]: DEBUG oslo_concurrency.lockutils [None req-59bbe742-6904-44d5-82f1-ce8e482a3a79 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71011) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 698.109730] env[71011]: DEBUG oslo_concurrency.lockutils [None req-59bbe742-6904-44d5-82f1-ce8e482a3a79 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71011) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 698.109969] env[71011]: DEBUG oslo_concurrency.lockutils [None req-59bbe742-6904-44d5-82f1-ce8e482a3a79 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71011) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 698.123105] env[71011]: DEBUG oslo_concurrency.lockutils [None req-59bbe742-6904-44d5-82f1-ce8e482a3a79 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71011) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 698.123105] env[71011]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-59bbe742-6904-44d5-82f1-ce8e482a3a79 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 698.123105] env[71011]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-59bbe742-6904-44d5-82f1-ce8e482a3a79 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=71011) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 700.124942] env[71011]: CRITICAL neutron [None req-59bbe742-6904-44d5-82f1-ce8e482a3a79 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 700.124942] env[71011]: ERROR neutron Traceback (most recent call last): [ 700.124942] env[71011]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 700.124942] env[71011]: ERROR neutron sys.exit(main()) [ 700.124942] env[71011]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 700.124942] env[71011]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 700.124942] env[71011]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 700.124942] env[71011]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 700.124942] env[71011]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 700.124942] env[71011]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 700.124942] env[71011]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 700.124942] env[71011]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 700.124942] env[71011]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 700.124942] env[71011]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 700.124942] env[71011]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 700.124942] env[71011]: ERROR neutron response = self.get(path=path, params=params) [ 700.125579] env[71011]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 700.125579] env[71011]: ERROR neutron raise RuntimeError(msg, last_err) [ 700.125579] env[71011]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 700.125579] env[71011]: ERROR neutron [ 702.847758] env[71149]: INFO neutron.common.config [-] Logging enabled! [ 702.848047] env[71149]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 702.848394] env[71149]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=71149) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 702.848637] env[71149]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 702.848740] env[71149]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 702.849583] env[71149]: WARNING oslo_policy.policy [None req-d6e8f906-e829-47fa-9f16-d0b9cf465aee None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 702.849791] env[71149]: WARNING oslo_policy.policy [None req-d6e8f906-e829-47fa-9f16-d0b9cf465aee None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 702.851044] env[71149]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-d6e8f906-e829-47fa-9f16-d0b9cf465aee None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 702.865356] env[71149]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d6e8f906-e829-47fa-9f16-d0b9cf465aee None None] Session token - acquiring [ 702.866162] env[71149]: DEBUG networking_nsxv3.common.locking [None req-d6e8f906-e829-47fa-9f16-d0b9cf465aee None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71149) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 702.866359] env[71149]: DEBUG oslo_concurrency.lockutils [None req-d6e8f906-e829-47fa-9f16-d0b9cf465aee None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71149) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 702.866567] env[71149]: DEBUG oslo_concurrency.lockutils [None req-d6e8f906-e829-47fa-9f16-d0b9cf465aee None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71149) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 702.866845] env[71149]: DEBUG oslo_concurrency.lockutils [None req-d6e8f906-e829-47fa-9f16-d0b9cf465aee None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71149) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 702.875920] env[71149]: DEBUG oslo_concurrency.lockutils [None req-d6e8f906-e829-47fa-9f16-d0b9cf465aee None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71149) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 702.876594] env[71149]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d6e8f906-e829-47fa-9f16-d0b9cf465aee None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 702.876594] env[71149]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d6e8f906-e829-47fa-9f16-d0b9cf465aee None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=71149) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 704.882431] env[71149]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d6e8f906-e829-47fa-9f16-d0b9cf465aee None None] Session token - acquiring [ 704.882746] env[71149]: DEBUG networking_nsxv3.common.locking [None req-d6e8f906-e829-47fa-9f16-d0b9cf465aee None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71149) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 704.882914] env[71149]: DEBUG oslo_concurrency.lockutils [None req-d6e8f906-e829-47fa-9f16-d0b9cf465aee None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71149) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 704.883048] env[71149]: DEBUG oslo_concurrency.lockutils [None req-d6e8f906-e829-47fa-9f16-d0b9cf465aee None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71149) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 704.883328] env[71149]: DEBUG oslo_concurrency.lockutils [None req-d6e8f906-e829-47fa-9f16-d0b9cf465aee None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71149) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 704.895653] env[71149]: DEBUG oslo_concurrency.lockutils [None req-d6e8f906-e829-47fa-9f16-d0b9cf465aee None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71149) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 704.895653] env[71149]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d6e8f906-e829-47fa-9f16-d0b9cf465aee None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 704.895653] env[71149]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d6e8f906-e829-47fa-9f16-d0b9cf465aee None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=71149) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 706.895851] env[71149]: CRITICAL neutron [None req-d6e8f906-e829-47fa-9f16-d0b9cf465aee None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 706.895851] env[71149]: ERROR neutron Traceback (most recent call last): [ 706.895851] env[71149]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 706.895851] env[71149]: ERROR neutron sys.exit(main()) [ 706.895851] env[71149]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 706.895851] env[71149]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 706.895851] env[71149]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 706.895851] env[71149]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 706.895851] env[71149]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 706.895851] env[71149]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 706.895851] env[71149]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 706.895851] env[71149]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 706.895851] env[71149]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 706.895851] env[71149]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 706.895851] env[71149]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 706.895851] env[71149]: ERROR neutron response = self.get(path=path, params=params) [ 706.897028] env[71149]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 706.897028] env[71149]: ERROR neutron raise RuntimeError(msg, last_err) [ 706.897028] env[71149]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 706.897028] env[71149]: ERROR neutron [ 709.731112] env[71384]: INFO neutron.common.config [-] Logging enabled! [ 709.731450] env[71384]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 709.731873] env[71384]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=71384) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 709.731873] env[71384]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 709.731965] env[71384]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 709.732803] env[71384]: WARNING oslo_policy.policy [None req-d98b1da4-339e-42d5-8f3f-d5f9f77def6e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 709.733014] env[71384]: WARNING oslo_policy.policy [None req-d98b1da4-339e-42d5-8f3f-d5f9f77def6e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 709.734253] env[71384]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-d98b1da4-339e-42d5-8f3f-d5f9f77def6e None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 709.748284] env[71384]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d98b1da4-339e-42d5-8f3f-d5f9f77def6e None None] Session token - acquiring [ 709.749073] env[71384]: DEBUG networking_nsxv3.common.locking [None req-d98b1da4-339e-42d5-8f3f-d5f9f77def6e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71384) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 709.749260] env[71384]: DEBUG oslo_concurrency.lockutils [None req-d98b1da4-339e-42d5-8f3f-d5f9f77def6e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71384) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 709.749389] env[71384]: DEBUG oslo_concurrency.lockutils [None req-d98b1da4-339e-42d5-8f3f-d5f9f77def6e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71384) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 709.749755] env[71384]: DEBUG oslo_concurrency.lockutils [None req-d98b1da4-339e-42d5-8f3f-d5f9f77def6e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71384) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 709.760831] env[71384]: DEBUG oslo_concurrency.lockutils [None req-d98b1da4-339e-42d5-8f3f-d5f9f77def6e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71384) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 709.761094] env[71384]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d98b1da4-339e-42d5-8f3f-d5f9f77def6e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 709.761256] env[71384]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d98b1da4-339e-42d5-8f3f-d5f9f77def6e None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=71384) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 711.767599] env[71384]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d98b1da4-339e-42d5-8f3f-d5f9f77def6e None None] Session token - acquiring [ 711.768019] env[71384]: DEBUG networking_nsxv3.common.locking [None req-d98b1da4-339e-42d5-8f3f-d5f9f77def6e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71384) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 711.768176] env[71384]: DEBUG oslo_concurrency.lockutils [None req-d98b1da4-339e-42d5-8f3f-d5f9f77def6e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71384) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 711.768307] env[71384]: DEBUG oslo_concurrency.lockutils [None req-d98b1da4-339e-42d5-8f3f-d5f9f77def6e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71384) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 711.768580] env[71384]: DEBUG oslo_concurrency.lockutils [None req-d98b1da4-339e-42d5-8f3f-d5f9f77def6e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71384) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 711.778112] env[71384]: DEBUG oslo_concurrency.lockutils [None req-d98b1da4-339e-42d5-8f3f-d5f9f77def6e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71384) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 711.778361] env[71384]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d98b1da4-339e-42d5-8f3f-d5f9f77def6e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 711.778500] env[71384]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d98b1da4-339e-42d5-8f3f-d5f9f77def6e None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=71384) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 713.780920] env[71384]: CRITICAL neutron [None req-d98b1da4-339e-42d5-8f3f-d5f9f77def6e None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 713.780920] env[71384]: ERROR neutron Traceback (most recent call last): [ 713.780920] env[71384]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 713.780920] env[71384]: ERROR neutron sys.exit(main()) [ 713.780920] env[71384]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 713.780920] env[71384]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 713.780920] env[71384]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 713.780920] env[71384]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 713.780920] env[71384]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 713.780920] env[71384]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 713.780920] env[71384]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 713.780920] env[71384]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 713.780920] env[71384]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 713.780920] env[71384]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 713.780920] env[71384]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 713.780920] env[71384]: ERROR neutron response = self.get(path=path, params=params) [ 713.781579] env[71384]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 713.781579] env[71384]: ERROR neutron raise RuntimeError(msg, last_err) [ 713.781579] env[71384]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 713.781579] env[71384]: ERROR neutron [ 716.415969] env[71616]: INFO neutron.common.config [-] Logging enabled! [ 716.416398] env[71616]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 716.416546] env[71616]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=71616) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 716.416781] env[71616]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 716.416877] env[71616]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 716.417713] env[71616]: WARNING oslo_policy.policy [None req-2465b0ba-0bc3-4f9f-a56e-a20414f3a48e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 716.418121] env[71616]: WARNING oslo_policy.policy [None req-2465b0ba-0bc3-4f9f-a56e-a20414f3a48e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 716.419258] env[71616]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-2465b0ba-0bc3-4f9f-a56e-a20414f3a48e None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 716.434294] env[71616]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2465b0ba-0bc3-4f9f-a56e-a20414f3a48e None None] Session token - acquiring [ 716.435172] env[71616]: DEBUG networking_nsxv3.common.locking [None req-2465b0ba-0bc3-4f9f-a56e-a20414f3a48e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71616) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 716.435372] env[71616]: DEBUG oslo_concurrency.lockutils [None req-2465b0ba-0bc3-4f9f-a56e-a20414f3a48e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71616) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 716.435496] env[71616]: DEBUG oslo_concurrency.lockutils [None req-2465b0ba-0bc3-4f9f-a56e-a20414f3a48e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71616) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 716.435864] env[71616]: DEBUG oslo_concurrency.lockutils [None req-2465b0ba-0bc3-4f9f-a56e-a20414f3a48e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71616) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 716.446008] env[71616]: DEBUG oslo_concurrency.lockutils [None req-2465b0ba-0bc3-4f9f-a56e-a20414f3a48e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71616) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 716.446293] env[71616]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2465b0ba-0bc3-4f9f-a56e-a20414f3a48e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 716.446443] env[71616]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2465b0ba-0bc3-4f9f-a56e-a20414f3a48e None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=71616) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 718.452606] env[71616]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2465b0ba-0bc3-4f9f-a56e-a20414f3a48e None None] Session token - acquiring [ 718.452908] env[71616]: DEBUG networking_nsxv3.common.locking [None req-2465b0ba-0bc3-4f9f-a56e-a20414f3a48e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71616) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 718.453099] env[71616]: DEBUG oslo_concurrency.lockutils [None req-2465b0ba-0bc3-4f9f-a56e-a20414f3a48e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71616) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 718.453231] env[71616]: DEBUG oslo_concurrency.lockutils [None req-2465b0ba-0bc3-4f9f-a56e-a20414f3a48e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71616) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 718.453517] env[71616]: DEBUG oslo_concurrency.lockutils [None req-2465b0ba-0bc3-4f9f-a56e-a20414f3a48e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71616) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 718.463595] env[71616]: DEBUG oslo_concurrency.lockutils [None req-2465b0ba-0bc3-4f9f-a56e-a20414f3a48e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71616) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 718.463900] env[71616]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2465b0ba-0bc3-4f9f-a56e-a20414f3a48e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 718.464063] env[71616]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2465b0ba-0bc3-4f9f-a56e-a20414f3a48e None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=71616) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 720.466979] env[71616]: CRITICAL neutron [None req-2465b0ba-0bc3-4f9f-a56e-a20414f3a48e None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 720.466979] env[71616]: ERROR neutron Traceback (most recent call last): [ 720.466979] env[71616]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 720.466979] env[71616]: ERROR neutron sys.exit(main()) [ 720.466979] env[71616]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 720.466979] env[71616]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 720.466979] env[71616]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 720.466979] env[71616]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 720.466979] env[71616]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 720.466979] env[71616]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 720.466979] env[71616]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 720.466979] env[71616]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 720.466979] env[71616]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 720.466979] env[71616]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 720.466979] env[71616]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 720.466979] env[71616]: ERROR neutron response = self.get(path=path, params=params) [ 720.467857] env[71616]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 720.467857] env[71616]: ERROR neutron raise RuntimeError(msg, last_err) [ 720.467857] env[71616]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 720.467857] env[71616]: ERROR neutron [ 723.309905] env[71813]: INFO neutron.common.config [-] Logging enabled! [ 723.310137] env[71813]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 723.310460] env[71813]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=71813) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 723.310692] env[71813]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 723.310788] env[71813]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 723.311691] env[71813]: WARNING oslo_policy.policy [None req-af90fd99-702c-42a2-bd55-b8b095542b10 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 723.311915] env[71813]: WARNING oslo_policy.policy [None req-af90fd99-702c-42a2-bd55-b8b095542b10 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 723.313203] env[71813]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-af90fd99-702c-42a2-bd55-b8b095542b10 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 723.328233] env[71813]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-af90fd99-702c-42a2-bd55-b8b095542b10 None None] Session token - acquiring [ 723.329091] env[71813]: DEBUG networking_nsxv3.common.locking [None req-af90fd99-702c-42a2-bd55-b8b095542b10 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71813) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 723.329290] env[71813]: DEBUG oslo_concurrency.lockutils [None req-af90fd99-702c-42a2-bd55-b8b095542b10 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71813) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 723.329413] env[71813]: DEBUG oslo_concurrency.lockutils [None req-af90fd99-702c-42a2-bd55-b8b095542b10 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71813) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 723.329794] env[71813]: DEBUG oslo_concurrency.lockutils [None req-af90fd99-702c-42a2-bd55-b8b095542b10 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71813) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 723.345864] env[71813]: DEBUG oslo_concurrency.lockutils [None req-af90fd99-702c-42a2-bd55-b8b095542b10 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71813) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 723.346273] env[71813]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-af90fd99-702c-42a2-bd55-b8b095542b10 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 723.346330] env[71813]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-af90fd99-702c-42a2-bd55-b8b095542b10 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=71813) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 725.355176] env[71813]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-af90fd99-702c-42a2-bd55-b8b095542b10 None None] Session token - acquiring [ 725.355176] env[71813]: DEBUG networking_nsxv3.common.locking [None req-af90fd99-702c-42a2-bd55-b8b095542b10 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71813) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 725.355176] env[71813]: DEBUG oslo_concurrency.lockutils [None req-af90fd99-702c-42a2-bd55-b8b095542b10 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71813) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 725.355585] env[71813]: DEBUG oslo_concurrency.lockutils [None req-af90fd99-702c-42a2-bd55-b8b095542b10 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71813) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 725.355585] env[71813]: DEBUG oslo_concurrency.lockutils [None req-af90fd99-702c-42a2-bd55-b8b095542b10 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71813) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 725.366857] env[71813]: DEBUG oslo_concurrency.lockutils [None req-af90fd99-702c-42a2-bd55-b8b095542b10 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71813) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 725.367486] env[71813]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-af90fd99-702c-42a2-bd55-b8b095542b10 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 725.368043] env[71813]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-af90fd99-702c-42a2-bd55-b8b095542b10 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=71813) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 727.372097] env[71813]: CRITICAL neutron [None req-af90fd99-702c-42a2-bd55-b8b095542b10 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 727.372097] env[71813]: ERROR neutron Traceback (most recent call last): [ 727.372097] env[71813]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 727.372097] env[71813]: ERROR neutron sys.exit(main()) [ 727.372097] env[71813]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 727.372097] env[71813]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 727.372097] env[71813]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 727.372097] env[71813]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 727.372097] env[71813]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 727.372097] env[71813]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 727.372097] env[71813]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 727.372097] env[71813]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 727.372097] env[71813]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 727.372097] env[71813]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 727.372097] env[71813]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 727.372097] env[71813]: ERROR neutron response = self.get(path=path, params=params) [ 727.372929] env[71813]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 727.372929] env[71813]: ERROR neutron raise RuntimeError(msg, last_err) [ 727.372929] env[71813]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 727.372929] env[71813]: ERROR neutron [ 730.154256] env[72213]: INFO neutron.common.config [-] Logging enabled! [ 730.154510] env[72213]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 730.154801] env[72213]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=72213) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 730.155033] env[72213]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 730.155126] env[72213]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 730.156693] env[72213]: WARNING oslo_policy.policy [None req-fb478ba9-8a6f-43ae-8f8e-76e7e8041f26 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 730.156693] env[72213]: WARNING oslo_policy.policy [None req-fb478ba9-8a6f-43ae-8f8e-76e7e8041f26 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 730.157391] env[72213]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-fb478ba9-8a6f-43ae-8f8e-76e7e8041f26 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 730.171946] env[72213]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fb478ba9-8a6f-43ae-8f8e-76e7e8041f26 None None] Session token - acquiring [ 730.172805] env[72213]: DEBUG networking_nsxv3.common.locking [None req-fb478ba9-8a6f-43ae-8f8e-76e7e8041f26 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72213) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 730.173019] env[72213]: DEBUG oslo_concurrency.lockutils [None req-fb478ba9-8a6f-43ae-8f8e-76e7e8041f26 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72213) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 730.173145] env[72213]: DEBUG oslo_concurrency.lockutils [None req-fb478ba9-8a6f-43ae-8f8e-76e7e8041f26 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72213) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 730.173508] env[72213]: DEBUG oslo_concurrency.lockutils [None req-fb478ba9-8a6f-43ae-8f8e-76e7e8041f26 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72213) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 730.182796] env[72213]: DEBUG oslo_concurrency.lockutils [None req-fb478ba9-8a6f-43ae-8f8e-76e7e8041f26 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72213) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 730.183050] env[72213]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fb478ba9-8a6f-43ae-8f8e-76e7e8041f26 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 730.183195] env[72213]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fb478ba9-8a6f-43ae-8f8e-76e7e8041f26 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=72213) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 732.189219] env[72213]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fb478ba9-8a6f-43ae-8f8e-76e7e8041f26 None None] Session token - acquiring [ 732.189576] env[72213]: DEBUG networking_nsxv3.common.locking [None req-fb478ba9-8a6f-43ae-8f8e-76e7e8041f26 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72213) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 732.189726] env[72213]: DEBUG oslo_concurrency.lockutils [None req-fb478ba9-8a6f-43ae-8f8e-76e7e8041f26 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72213) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 732.189797] env[72213]: DEBUG oslo_concurrency.lockutils [None req-fb478ba9-8a6f-43ae-8f8e-76e7e8041f26 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72213) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 732.190076] env[72213]: DEBUG oslo_concurrency.lockutils [None req-fb478ba9-8a6f-43ae-8f8e-76e7e8041f26 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72213) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 732.199746] env[72213]: DEBUG oslo_concurrency.lockutils [None req-fb478ba9-8a6f-43ae-8f8e-76e7e8041f26 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72213) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 732.200024] env[72213]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fb478ba9-8a6f-43ae-8f8e-76e7e8041f26 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 732.200170] env[72213]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fb478ba9-8a6f-43ae-8f8e-76e7e8041f26 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=72213) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 734.212788] env[72213]: CRITICAL neutron [None req-fb478ba9-8a6f-43ae-8f8e-76e7e8041f26 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 734.212788] env[72213]: ERROR neutron Traceback (most recent call last): [ 734.212788] env[72213]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 734.212788] env[72213]: ERROR neutron sys.exit(main()) [ 734.212788] env[72213]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 734.212788] env[72213]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 734.212788] env[72213]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 734.212788] env[72213]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 734.212788] env[72213]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 734.212788] env[72213]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 734.212788] env[72213]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 734.212788] env[72213]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 734.212788] env[72213]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 734.212788] env[72213]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 734.212788] env[72213]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 734.212788] env[72213]: ERROR neutron response = self.get(path=path, params=params) [ 734.213630] env[72213]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 734.213630] env[72213]: ERROR neutron raise RuntimeError(msg, last_err) [ 734.213630] env[72213]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 734.213630] env[72213]: ERROR neutron [ 737.030254] env[72523]: INFO neutron.common.config [-] Logging enabled! [ 737.030596] env[72523]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 737.030860] env[72523]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=72523) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 737.031104] env[72523]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 737.031201] env[72523]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 737.032103] env[72523]: WARNING oslo_policy.policy [None req-b3ac671e-3aad-4186-ac0d-75dbaf5b9d30 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 737.032335] env[72523]: WARNING oslo_policy.policy [None req-b3ac671e-3aad-4186-ac0d-75dbaf5b9d30 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 737.033558] env[72523]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-b3ac671e-3aad-4186-ac0d-75dbaf5b9d30 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 737.049607] env[72523]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b3ac671e-3aad-4186-ac0d-75dbaf5b9d30 None None] Session token - acquiring [ 737.050502] env[72523]: DEBUG networking_nsxv3.common.locking [None req-b3ac671e-3aad-4186-ac0d-75dbaf5b9d30 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72523) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 737.050716] env[72523]: DEBUG oslo_concurrency.lockutils [None req-b3ac671e-3aad-4186-ac0d-75dbaf5b9d30 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72523) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 737.050844] env[72523]: DEBUG oslo_concurrency.lockutils [None req-b3ac671e-3aad-4186-ac0d-75dbaf5b9d30 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72523) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 737.051217] env[72523]: DEBUG oslo_concurrency.lockutils [None req-b3ac671e-3aad-4186-ac0d-75dbaf5b9d30 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72523) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 737.060607] env[72523]: DEBUG oslo_concurrency.lockutils [None req-b3ac671e-3aad-4186-ac0d-75dbaf5b9d30 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72523) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 737.060863] env[72523]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b3ac671e-3aad-4186-ac0d-75dbaf5b9d30 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 737.061116] env[72523]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b3ac671e-3aad-4186-ac0d-75dbaf5b9d30 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=72523) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 739.067370] env[72523]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b3ac671e-3aad-4186-ac0d-75dbaf5b9d30 None None] Session token - acquiring [ 739.067819] env[72523]: DEBUG networking_nsxv3.common.locking [None req-b3ac671e-3aad-4186-ac0d-75dbaf5b9d30 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72523) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 739.068035] env[72523]: DEBUG oslo_concurrency.lockutils [None req-b3ac671e-3aad-4186-ac0d-75dbaf5b9d30 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72523) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 739.068169] env[72523]: DEBUG oslo_concurrency.lockutils [None req-b3ac671e-3aad-4186-ac0d-75dbaf5b9d30 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72523) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 739.068456] env[72523]: DEBUG oslo_concurrency.lockutils [None req-b3ac671e-3aad-4186-ac0d-75dbaf5b9d30 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72523) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 739.077940] env[72523]: DEBUG oslo_concurrency.lockutils [None req-b3ac671e-3aad-4186-ac0d-75dbaf5b9d30 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72523) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 739.078194] env[72523]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b3ac671e-3aad-4186-ac0d-75dbaf5b9d30 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 739.078337] env[72523]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b3ac671e-3aad-4186-ac0d-75dbaf5b9d30 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=72523) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 741.081259] env[72523]: CRITICAL neutron [None req-b3ac671e-3aad-4186-ac0d-75dbaf5b9d30 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 741.081259] env[72523]: ERROR neutron Traceback (most recent call last): [ 741.081259] env[72523]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 741.081259] env[72523]: ERROR neutron sys.exit(main()) [ 741.081259] env[72523]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 741.081259] env[72523]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 741.081259] env[72523]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 741.081259] env[72523]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 741.081259] env[72523]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 741.081259] env[72523]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 741.081259] env[72523]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 741.081259] env[72523]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 741.081259] env[72523]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 741.081259] env[72523]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 741.081259] env[72523]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 741.081259] env[72523]: ERROR neutron response = self.get(path=path, params=params) [ 741.081793] env[72523]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 741.081793] env[72523]: ERROR neutron raise RuntimeError(msg, last_err) [ 741.081793] env[72523]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 741.081793] env[72523]: ERROR neutron [ 743.966616] env[72931]: INFO neutron.common.config [-] Logging enabled! [ 743.966860] env[72931]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 743.967146] env[72931]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=72931) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 743.967368] env[72931]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 743.967463] env[72931]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 743.968331] env[72931]: WARNING oslo_policy.policy [None req-fd45943b-7174-4c81-90f4-84f336d21a54 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 743.968549] env[72931]: WARNING oslo_policy.policy [None req-fd45943b-7174-4c81-90f4-84f336d21a54 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 743.969756] env[72931]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-fd45943b-7174-4c81-90f4-84f336d21a54 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 743.984048] env[72931]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fd45943b-7174-4c81-90f4-84f336d21a54 None None] Session token - acquiring [ 743.984900] env[72931]: DEBUG networking_nsxv3.common.locking [None req-fd45943b-7174-4c81-90f4-84f336d21a54 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72931) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 743.985108] env[72931]: DEBUG oslo_concurrency.lockutils [None req-fd45943b-7174-4c81-90f4-84f336d21a54 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72931) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 743.985237] env[72931]: DEBUG oslo_concurrency.lockutils [None req-fd45943b-7174-4c81-90f4-84f336d21a54 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72931) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 743.985593] env[72931]: DEBUG oslo_concurrency.lockutils [None req-fd45943b-7174-4c81-90f4-84f336d21a54 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72931) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 743.994330] env[72931]: DEBUG oslo_concurrency.lockutils [None req-fd45943b-7174-4c81-90f4-84f336d21a54 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72931) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 743.994569] env[72931]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fd45943b-7174-4c81-90f4-84f336d21a54 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 743.994717] env[72931]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fd45943b-7174-4c81-90f4-84f336d21a54 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=72931) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 746.000639] env[72931]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fd45943b-7174-4c81-90f4-84f336d21a54 None None] Session token - acquiring [ 746.000938] env[72931]: DEBUG networking_nsxv3.common.locking [None req-fd45943b-7174-4c81-90f4-84f336d21a54 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72931) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 746.001141] env[72931]: DEBUG oslo_concurrency.lockutils [None req-fd45943b-7174-4c81-90f4-84f336d21a54 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72931) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 746.001262] env[72931]: DEBUG oslo_concurrency.lockutils [None req-fd45943b-7174-4c81-90f4-84f336d21a54 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72931) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 746.001552] env[72931]: DEBUG oslo_concurrency.lockutils [None req-fd45943b-7174-4c81-90f4-84f336d21a54 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72931) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 746.010983] env[72931]: DEBUG oslo_concurrency.lockutils [None req-fd45943b-7174-4c81-90f4-84f336d21a54 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72931) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 746.011065] env[72931]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fd45943b-7174-4c81-90f4-84f336d21a54 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 746.011170] env[72931]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-fd45943b-7174-4c81-90f4-84f336d21a54 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=72931) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 748.012890] env[72931]: CRITICAL neutron [None req-fd45943b-7174-4c81-90f4-84f336d21a54 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 748.012890] env[72931]: ERROR neutron Traceback (most recent call last): [ 748.012890] env[72931]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 748.012890] env[72931]: ERROR neutron sys.exit(main()) [ 748.012890] env[72931]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 748.012890] env[72931]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 748.012890] env[72931]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 748.012890] env[72931]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 748.012890] env[72931]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 748.012890] env[72931]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 748.012890] env[72931]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 748.012890] env[72931]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 748.012890] env[72931]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 748.012890] env[72931]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 748.012890] env[72931]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 748.012890] env[72931]: ERROR neutron response = self.get(path=path, params=params) [ 748.013907] env[72931]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 748.013907] env[72931]: ERROR neutron raise RuntimeError(msg, last_err) [ 748.013907] env[72931]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 748.013907] env[72931]: ERROR neutron [ 750.590193] env[73177]: INFO neutron.common.config [-] Logging enabled! [ 750.590485] env[73177]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 750.590773] env[73177]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73177) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 750.591016] env[73177]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 750.591110] env[73177]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 750.592486] env[73177]: WARNING oslo_policy.policy [None req-0a3ca6a5-a515-48f2-be68-1a291c2dd1db None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 750.592486] env[73177]: WARNING oslo_policy.policy [None req-0a3ca6a5-a515-48f2-be68-1a291c2dd1db None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 750.593480] env[73177]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-0a3ca6a5-a515-48f2-be68-1a291c2dd1db None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 750.609183] env[73177]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0a3ca6a5-a515-48f2-be68-1a291c2dd1db None None] Session token - acquiring [ 750.611453] env[73177]: DEBUG networking_nsxv3.common.locking [None req-0a3ca6a5-a515-48f2-be68-1a291c2dd1db None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73177) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 750.611663] env[73177]: DEBUG oslo_concurrency.lockutils [None req-0a3ca6a5-a515-48f2-be68-1a291c2dd1db None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73177) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 750.611788] env[73177]: DEBUG oslo_concurrency.lockutils [None req-0a3ca6a5-a515-48f2-be68-1a291c2dd1db None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73177) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 750.612175] env[73177]: DEBUG oslo_concurrency.lockutils [None req-0a3ca6a5-a515-48f2-be68-1a291c2dd1db None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73177) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 750.622175] env[73177]: DEBUG oslo_concurrency.lockutils [None req-0a3ca6a5-a515-48f2-be68-1a291c2dd1db None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73177) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 750.622418] env[73177]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0a3ca6a5-a515-48f2-be68-1a291c2dd1db None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 750.622558] env[73177]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0a3ca6a5-a515-48f2-be68-1a291c2dd1db None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73177) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 752.628591] env[73177]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0a3ca6a5-a515-48f2-be68-1a291c2dd1db None None] Session token - acquiring [ 752.628946] env[73177]: DEBUG networking_nsxv3.common.locking [None req-0a3ca6a5-a515-48f2-be68-1a291c2dd1db None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73177) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 752.629067] env[73177]: DEBUG oslo_concurrency.lockutils [None req-0a3ca6a5-a515-48f2-be68-1a291c2dd1db None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73177) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 752.629192] env[73177]: DEBUG oslo_concurrency.lockutils [None req-0a3ca6a5-a515-48f2-be68-1a291c2dd1db None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73177) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 752.629480] env[73177]: DEBUG oslo_concurrency.lockutils [None req-0a3ca6a5-a515-48f2-be68-1a291c2dd1db None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73177) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 752.638607] env[73177]: DEBUG oslo_concurrency.lockutils [None req-0a3ca6a5-a515-48f2-be68-1a291c2dd1db None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73177) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 752.638834] env[73177]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0a3ca6a5-a515-48f2-be68-1a291c2dd1db None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 752.638977] env[73177]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0a3ca6a5-a515-48f2-be68-1a291c2dd1db None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73177) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 754.641963] env[73177]: CRITICAL neutron [None req-0a3ca6a5-a515-48f2-be68-1a291c2dd1db None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 754.641963] env[73177]: ERROR neutron Traceback (most recent call last): [ 754.641963] env[73177]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 754.641963] env[73177]: ERROR neutron sys.exit(main()) [ 754.641963] env[73177]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 754.641963] env[73177]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 754.641963] env[73177]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 754.641963] env[73177]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 754.641963] env[73177]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 754.641963] env[73177]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 754.641963] env[73177]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 754.641963] env[73177]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 754.641963] env[73177]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 754.641963] env[73177]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 754.641963] env[73177]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 754.641963] env[73177]: ERROR neutron response = self.get(path=path, params=params) [ 754.642614] env[73177]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 754.642614] env[73177]: ERROR neutron raise RuntimeError(msg, last_err) [ 754.642614] env[73177]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 754.642614] env[73177]: ERROR neutron [ 757.129774] env[73298]: INFO neutron.common.config [-] Logging enabled! [ 757.130124] env[73298]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 757.130339] env[73298]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73298) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 757.130575] env[73298]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 757.130678] env[73298]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 757.131514] env[73298]: WARNING oslo_policy.policy [None req-63753e28-ea29-4ded-b0ed-4aad81b4e66b None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 757.131809] env[73298]: WARNING oslo_policy.policy [None req-63753e28-ea29-4ded-b0ed-4aad81b4e66b None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 757.132985] env[73298]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-63753e28-ea29-4ded-b0ed-4aad81b4e66b None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 757.148431] env[73298]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63753e28-ea29-4ded-b0ed-4aad81b4e66b None None] Session token - acquiring [ 757.149289] env[73298]: DEBUG networking_nsxv3.common.locking [None req-63753e28-ea29-4ded-b0ed-4aad81b4e66b None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73298) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 757.149499] env[73298]: DEBUG oslo_concurrency.lockutils [None req-63753e28-ea29-4ded-b0ed-4aad81b4e66b None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73298) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 757.149626] env[73298]: DEBUG oslo_concurrency.lockutils [None req-63753e28-ea29-4ded-b0ed-4aad81b4e66b None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73298) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 757.150000] env[73298]: DEBUG oslo_concurrency.lockutils [None req-63753e28-ea29-4ded-b0ed-4aad81b4e66b None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73298) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 757.159286] env[73298]: DEBUG oslo_concurrency.lockutils [None req-63753e28-ea29-4ded-b0ed-4aad81b4e66b None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73298) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 757.159551] env[73298]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63753e28-ea29-4ded-b0ed-4aad81b4e66b None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 757.159700] env[73298]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63753e28-ea29-4ded-b0ed-4aad81b4e66b None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73298) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 759.166596] env[73298]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63753e28-ea29-4ded-b0ed-4aad81b4e66b None None] Session token - acquiring [ 759.166983] env[73298]: DEBUG networking_nsxv3.common.locking [None req-63753e28-ea29-4ded-b0ed-4aad81b4e66b None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73298) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 759.167158] env[73298]: DEBUG oslo_concurrency.lockutils [None req-63753e28-ea29-4ded-b0ed-4aad81b4e66b None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73298) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 759.167278] env[73298]: DEBUG oslo_concurrency.lockutils [None req-63753e28-ea29-4ded-b0ed-4aad81b4e66b None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73298) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 759.167551] env[73298]: DEBUG oslo_concurrency.lockutils [None req-63753e28-ea29-4ded-b0ed-4aad81b4e66b None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73298) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 759.177694] env[73298]: DEBUG oslo_concurrency.lockutils [None req-63753e28-ea29-4ded-b0ed-4aad81b4e66b None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73298) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 759.177936] env[73298]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63753e28-ea29-4ded-b0ed-4aad81b4e66b None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 759.178079] env[73298]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63753e28-ea29-4ded-b0ed-4aad81b4e66b None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73298) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 761.181130] env[73298]: CRITICAL neutron [None req-63753e28-ea29-4ded-b0ed-4aad81b4e66b None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 761.181130] env[73298]: ERROR neutron Traceback (most recent call last): [ 761.181130] env[73298]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 761.181130] env[73298]: ERROR neutron sys.exit(main()) [ 761.181130] env[73298]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 761.181130] env[73298]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 761.181130] env[73298]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 761.181130] env[73298]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 761.181130] env[73298]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 761.181130] env[73298]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 761.181130] env[73298]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 761.181130] env[73298]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 761.181130] env[73298]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 761.181130] env[73298]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 761.181130] env[73298]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 761.181130] env[73298]: ERROR neutron response = self.get(path=path, params=params) [ 761.181740] env[73298]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 761.181740] env[73298]: ERROR neutron raise RuntimeError(msg, last_err) [ 761.181740] env[73298]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 761.181740] env[73298]: ERROR neutron [ 763.974428] env[73415]: INFO neutron.common.config [-] Logging enabled! [ 763.974852] env[73415]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 763.975026] env[73415]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73415) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 763.975224] env[73415]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 763.975316] env[73415]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 763.976482] env[73415]: WARNING oslo_policy.policy [None req-9b635435-1053-49c9-a933-9480d3b32035 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 763.976769] env[73415]: WARNING oslo_policy.policy [None req-9b635435-1053-49c9-a933-9480d3b32035 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 763.978084] env[73415]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-9b635435-1053-49c9-a933-9480d3b32035 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 763.995483] env[73415]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b635435-1053-49c9-a933-9480d3b32035 None None] Session token - acquiring [ 763.996342] env[73415]: DEBUG networking_nsxv3.common.locking [None req-9b635435-1053-49c9-a933-9480d3b32035 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73415) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 763.996546] env[73415]: DEBUG oslo_concurrency.lockutils [None req-9b635435-1053-49c9-a933-9480d3b32035 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73415) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 763.996676] env[73415]: DEBUG oslo_concurrency.lockutils [None req-9b635435-1053-49c9-a933-9480d3b32035 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73415) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 763.997077] env[73415]: DEBUG oslo_concurrency.lockutils [None req-9b635435-1053-49c9-a933-9480d3b32035 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73415) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 764.007461] env[73415]: DEBUG oslo_concurrency.lockutils [None req-9b635435-1053-49c9-a933-9480d3b32035 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73415) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 764.007723] env[73415]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b635435-1053-49c9-a933-9480d3b32035 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 764.007872] env[73415]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b635435-1053-49c9-a933-9480d3b32035 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73415) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 766.014009] env[73415]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b635435-1053-49c9-a933-9480d3b32035 None None] Session token - acquiring [ 766.014348] env[73415]: DEBUG networking_nsxv3.common.locking [None req-9b635435-1053-49c9-a933-9480d3b32035 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73415) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 766.014547] env[73415]: DEBUG oslo_concurrency.lockutils [None req-9b635435-1053-49c9-a933-9480d3b32035 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73415) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 766.014675] env[73415]: DEBUG oslo_concurrency.lockutils [None req-9b635435-1053-49c9-a933-9480d3b32035 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73415) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 766.014954] env[73415]: DEBUG oslo_concurrency.lockutils [None req-9b635435-1053-49c9-a933-9480d3b32035 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73415) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 766.024459] env[73415]: DEBUG oslo_concurrency.lockutils [None req-9b635435-1053-49c9-a933-9480d3b32035 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73415) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 766.024671] env[73415]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b635435-1053-49c9-a933-9480d3b32035 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 766.024839] env[73415]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9b635435-1053-49c9-a933-9480d3b32035 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73415) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 768.027803] env[73415]: CRITICAL neutron [None req-9b635435-1053-49c9-a933-9480d3b32035 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 768.027803] env[73415]: ERROR neutron Traceback (most recent call last): [ 768.027803] env[73415]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 768.027803] env[73415]: ERROR neutron sys.exit(main()) [ 768.027803] env[73415]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 768.027803] env[73415]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 768.027803] env[73415]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 768.027803] env[73415]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 768.027803] env[73415]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 768.027803] env[73415]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 768.027803] env[73415]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 768.027803] env[73415]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 768.027803] env[73415]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 768.027803] env[73415]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 768.027803] env[73415]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 768.027803] env[73415]: ERROR neutron response = self.get(path=path, params=params) [ 768.028473] env[73415]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 768.028473] env[73415]: ERROR neutron raise RuntimeError(msg, last_err) [ 768.028473] env[73415]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 768.028473] env[73415]: ERROR neutron [ 770.626781] env[73530]: INFO neutron.common.config [-] Logging enabled! [ 770.627178] env[73530]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 770.627348] env[73530]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73530) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 770.627543] env[73530]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 770.627626] env[73530]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 770.628536] env[73530]: WARNING oslo_policy.policy [None req-5ad67a2a-37a7-43e2-b782-c4ba134b5cad None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 770.628763] env[73530]: WARNING oslo_policy.policy [None req-5ad67a2a-37a7-43e2-b782-c4ba134b5cad None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 770.630010] env[73530]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-5ad67a2a-37a7-43e2-b782-c4ba134b5cad None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 770.645096] env[73530]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5ad67a2a-37a7-43e2-b782-c4ba134b5cad None None] Session token - acquiring [ 770.645993] env[73530]: DEBUG networking_nsxv3.common.locking [None req-5ad67a2a-37a7-43e2-b782-c4ba134b5cad None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73530) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 770.646374] env[73530]: DEBUG oslo_concurrency.lockutils [None req-5ad67a2a-37a7-43e2-b782-c4ba134b5cad None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73530) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 770.646608] env[73530]: DEBUG oslo_concurrency.lockutils [None req-5ad67a2a-37a7-43e2-b782-c4ba134b5cad None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73530) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 770.647146] env[73530]: DEBUG oslo_concurrency.lockutils [None req-5ad67a2a-37a7-43e2-b782-c4ba134b5cad None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73530) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 770.656402] env[73530]: DEBUG oslo_concurrency.lockutils [None req-5ad67a2a-37a7-43e2-b782-c4ba134b5cad None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73530) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 770.656852] env[73530]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5ad67a2a-37a7-43e2-b782-c4ba134b5cad None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 770.657096] env[73530]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5ad67a2a-37a7-43e2-b782-c4ba134b5cad None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73530) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 772.663235] env[73530]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5ad67a2a-37a7-43e2-b782-c4ba134b5cad None None] Session token - acquiring [ 772.663651] env[73530]: DEBUG networking_nsxv3.common.locking [None req-5ad67a2a-37a7-43e2-b782-c4ba134b5cad None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73530) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 772.663997] env[73530]: DEBUG oslo_concurrency.lockutils [None req-5ad67a2a-37a7-43e2-b782-c4ba134b5cad None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73530) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 772.664175] env[73530]: DEBUG oslo_concurrency.lockutils [None req-5ad67a2a-37a7-43e2-b782-c4ba134b5cad None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73530) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 772.664461] env[73530]: DEBUG oslo_concurrency.lockutils [None req-5ad67a2a-37a7-43e2-b782-c4ba134b5cad None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73530) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 772.673555] env[73530]: DEBUG oslo_concurrency.lockutils [None req-5ad67a2a-37a7-43e2-b782-c4ba134b5cad None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73530) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 772.673863] env[73530]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5ad67a2a-37a7-43e2-b782-c4ba134b5cad None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 772.674020] env[73530]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5ad67a2a-37a7-43e2-b782-c4ba134b5cad None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73530) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 774.676943] env[73530]: CRITICAL neutron [None req-5ad67a2a-37a7-43e2-b782-c4ba134b5cad None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 774.676943] env[73530]: ERROR neutron Traceback (most recent call last): [ 774.676943] env[73530]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 774.676943] env[73530]: ERROR neutron sys.exit(main()) [ 774.676943] env[73530]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 774.676943] env[73530]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 774.676943] env[73530]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 774.676943] env[73530]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 774.676943] env[73530]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 774.676943] env[73530]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 774.676943] env[73530]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 774.676943] env[73530]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 774.676943] env[73530]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 774.676943] env[73530]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 774.676943] env[73530]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 774.676943] env[73530]: ERROR neutron response = self.get(path=path, params=params) [ 774.677529] env[73530]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 774.677529] env[73530]: ERROR neutron raise RuntimeError(msg, last_err) [ 774.677529] env[73530]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 774.677529] env[73530]: ERROR neutron [ 777.132571] env[73601]: INFO neutron.common.config [-] Logging enabled! [ 777.132897] env[73601]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 777.133164] env[73601]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73601) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 777.133300] env[73601]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 777.133393] env[73601]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 777.134266] env[73601]: WARNING oslo_policy.policy [None req-8df74cfe-c61e-4221-9ec5-7c600efad761 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 777.134471] env[73601]: WARNING oslo_policy.policy [None req-8df74cfe-c61e-4221-9ec5-7c600efad761 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 777.135702] env[73601]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-8df74cfe-c61e-4221-9ec5-7c600efad761 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 777.150713] env[73601]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8df74cfe-c61e-4221-9ec5-7c600efad761 None None] Session token - acquiring [ 777.151564] env[73601]: DEBUG networking_nsxv3.common.locking [None req-8df74cfe-c61e-4221-9ec5-7c600efad761 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73601) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 777.151810] env[73601]: DEBUG oslo_concurrency.lockutils [None req-8df74cfe-c61e-4221-9ec5-7c600efad761 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73601) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 777.151945] env[73601]: DEBUG oslo_concurrency.lockutils [None req-8df74cfe-c61e-4221-9ec5-7c600efad761 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73601) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 777.152334] env[73601]: DEBUG oslo_concurrency.lockutils [None req-8df74cfe-c61e-4221-9ec5-7c600efad761 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73601) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 777.162076] env[73601]: DEBUG oslo_concurrency.lockutils [None req-8df74cfe-c61e-4221-9ec5-7c600efad761 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73601) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 777.162197] env[73601]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8df74cfe-c61e-4221-9ec5-7c600efad761 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 777.162324] env[73601]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8df74cfe-c61e-4221-9ec5-7c600efad761 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73601) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 779.168501] env[73601]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8df74cfe-c61e-4221-9ec5-7c600efad761 None None] Session token - acquiring [ 779.168910] env[73601]: DEBUG networking_nsxv3.common.locking [None req-8df74cfe-c61e-4221-9ec5-7c600efad761 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73601) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 779.168978] env[73601]: DEBUG oslo_concurrency.lockutils [None req-8df74cfe-c61e-4221-9ec5-7c600efad761 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73601) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 779.169089] env[73601]: DEBUG oslo_concurrency.lockutils [None req-8df74cfe-c61e-4221-9ec5-7c600efad761 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73601) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 779.169374] env[73601]: DEBUG oslo_concurrency.lockutils [None req-8df74cfe-c61e-4221-9ec5-7c600efad761 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73601) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 779.180140] env[73601]: DEBUG oslo_concurrency.lockutils [None req-8df74cfe-c61e-4221-9ec5-7c600efad761 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73601) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 779.180439] env[73601]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8df74cfe-c61e-4221-9ec5-7c600efad761 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 779.180575] env[73601]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8df74cfe-c61e-4221-9ec5-7c600efad761 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73601) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 781.183512] env[73601]: CRITICAL neutron [None req-8df74cfe-c61e-4221-9ec5-7c600efad761 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 781.183512] env[73601]: ERROR neutron Traceback (most recent call last): [ 781.183512] env[73601]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 781.183512] env[73601]: ERROR neutron sys.exit(main()) [ 781.183512] env[73601]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 781.183512] env[73601]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 781.183512] env[73601]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 781.183512] env[73601]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 781.183512] env[73601]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 781.183512] env[73601]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 781.183512] env[73601]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 781.183512] env[73601]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 781.183512] env[73601]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 781.183512] env[73601]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 781.183512] env[73601]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 781.183512] env[73601]: ERROR neutron response = self.get(path=path, params=params) [ 781.184270] env[73601]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 781.184270] env[73601]: ERROR neutron raise RuntimeError(msg, last_err) [ 781.184270] env[73601]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 781.184270] env[73601]: ERROR neutron [ 783.939112] env[73803]: INFO neutron.common.config [-] Logging enabled! [ 783.939112] env[73803]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 783.939112] env[73803]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73803) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 783.939402] env[73803]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 783.939402] env[73803]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 783.940278] env[73803]: WARNING oslo_policy.policy [None req-34a7fc93-84ad-4f10-a962-38dfbf4a5353 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 783.940492] env[73803]: WARNING oslo_policy.policy [None req-34a7fc93-84ad-4f10-a962-38dfbf4a5353 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 783.941686] env[73803]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-34a7fc93-84ad-4f10-a962-38dfbf4a5353 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 783.955543] env[73803]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-34a7fc93-84ad-4f10-a962-38dfbf4a5353 None None] Session token - acquiring [ 783.956364] env[73803]: DEBUG networking_nsxv3.common.locking [None req-34a7fc93-84ad-4f10-a962-38dfbf4a5353 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73803) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 783.956579] env[73803]: DEBUG oslo_concurrency.lockutils [None req-34a7fc93-84ad-4f10-a962-38dfbf4a5353 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73803) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 783.956707] env[73803]: DEBUG oslo_concurrency.lockutils [None req-34a7fc93-84ad-4f10-a962-38dfbf4a5353 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73803) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 783.957082] env[73803]: DEBUG oslo_concurrency.lockutils [None req-34a7fc93-84ad-4f10-a962-38dfbf4a5353 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73803) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 783.966340] env[73803]: DEBUG oslo_concurrency.lockutils [None req-34a7fc93-84ad-4f10-a962-38dfbf4a5353 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73803) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 783.966602] env[73803]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-34a7fc93-84ad-4f10-a962-38dfbf4a5353 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 783.966748] env[73803]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-34a7fc93-84ad-4f10-a962-38dfbf4a5353 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73803) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 785.972893] env[73803]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-34a7fc93-84ad-4f10-a962-38dfbf4a5353 None None] Session token - acquiring [ 785.973228] env[73803]: DEBUG networking_nsxv3.common.locking [None req-34a7fc93-84ad-4f10-a962-38dfbf4a5353 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73803) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 785.973458] env[73803]: DEBUG oslo_concurrency.lockutils [None req-34a7fc93-84ad-4f10-a962-38dfbf4a5353 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73803) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 785.973548] env[73803]: DEBUG oslo_concurrency.lockutils [None req-34a7fc93-84ad-4f10-a962-38dfbf4a5353 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73803) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 785.973877] env[73803]: DEBUG oslo_concurrency.lockutils [None req-34a7fc93-84ad-4f10-a962-38dfbf4a5353 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73803) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 785.982881] env[73803]: DEBUG oslo_concurrency.lockutils [None req-34a7fc93-84ad-4f10-a962-38dfbf4a5353 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73803) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 785.983174] env[73803]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-34a7fc93-84ad-4f10-a962-38dfbf4a5353 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 785.983314] env[73803]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-34a7fc93-84ad-4f10-a962-38dfbf4a5353 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73803) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 787.984916] env[73803]: CRITICAL neutron [None req-34a7fc93-84ad-4f10-a962-38dfbf4a5353 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 787.984916] env[73803]: ERROR neutron Traceback (most recent call last): [ 787.984916] env[73803]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 787.984916] env[73803]: ERROR neutron sys.exit(main()) [ 787.984916] env[73803]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 787.984916] env[73803]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 787.984916] env[73803]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 787.984916] env[73803]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 787.984916] env[73803]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 787.984916] env[73803]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 787.984916] env[73803]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 787.984916] env[73803]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 787.984916] env[73803]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 787.984916] env[73803]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 787.984916] env[73803]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 787.984916] env[73803]: ERROR neutron response = self.get(path=path, params=params) [ 787.985474] env[73803]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 787.985474] env[73803]: ERROR neutron raise RuntimeError(msg, last_err) [ 787.985474] env[73803]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 787.985474] env[73803]: ERROR neutron [ 790.660853] env[73898]: INFO neutron.common.config [-] Logging enabled! [ 790.661222] env[73898]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 790.661368] env[73898]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73898) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 790.661603] env[73898]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 790.662133] env[73898]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 790.662645] env[73898]: WARNING oslo_policy.policy [None req-4d43515d-af75-4c49-b6e8-fc1a647739ab None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 790.662896] env[73898]: WARNING oslo_policy.policy [None req-4d43515d-af75-4c49-b6e8-fc1a647739ab None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 790.664399] env[73898]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-4d43515d-af75-4c49-b6e8-fc1a647739ab None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 790.678028] env[73898]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4d43515d-af75-4c49-b6e8-fc1a647739ab None None] Session token - acquiring [ 790.678866] env[73898]: DEBUG networking_nsxv3.common.locking [None req-4d43515d-af75-4c49-b6e8-fc1a647739ab None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73898) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 790.679067] env[73898]: DEBUG oslo_concurrency.lockutils [None req-4d43515d-af75-4c49-b6e8-fc1a647739ab None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73898) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 790.679190] env[73898]: DEBUG oslo_concurrency.lockutils [None req-4d43515d-af75-4c49-b6e8-fc1a647739ab None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73898) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 790.679558] env[73898]: DEBUG oslo_concurrency.lockutils [None req-4d43515d-af75-4c49-b6e8-fc1a647739ab None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73898) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 790.689968] env[73898]: DEBUG oslo_concurrency.lockutils [None req-4d43515d-af75-4c49-b6e8-fc1a647739ab None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73898) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 790.691271] env[73898]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4d43515d-af75-4c49-b6e8-fc1a647739ab None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 790.691271] env[73898]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4d43515d-af75-4c49-b6e8-fc1a647739ab None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73898) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 792.696708] env[73898]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4d43515d-af75-4c49-b6e8-fc1a647739ab None None] Session token - acquiring [ 792.697093] env[73898]: DEBUG networking_nsxv3.common.locking [None req-4d43515d-af75-4c49-b6e8-fc1a647739ab None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73898) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 792.697224] env[73898]: DEBUG oslo_concurrency.lockutils [None req-4d43515d-af75-4c49-b6e8-fc1a647739ab None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73898) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 792.697350] env[73898]: DEBUG oslo_concurrency.lockutils [None req-4d43515d-af75-4c49-b6e8-fc1a647739ab None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73898) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 792.697614] env[73898]: DEBUG oslo_concurrency.lockutils [None req-4d43515d-af75-4c49-b6e8-fc1a647739ab None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73898) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 792.707009] env[73898]: DEBUG oslo_concurrency.lockutils [None req-4d43515d-af75-4c49-b6e8-fc1a647739ab None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73898) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 792.707303] env[73898]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4d43515d-af75-4c49-b6e8-fc1a647739ab None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 792.707457] env[73898]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4d43515d-af75-4c49-b6e8-fc1a647739ab None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73898) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 794.708894] env[73898]: CRITICAL neutron [None req-4d43515d-af75-4c49-b6e8-fc1a647739ab None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 794.708894] env[73898]: ERROR neutron Traceback (most recent call last): [ 794.708894] env[73898]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 794.708894] env[73898]: ERROR neutron sys.exit(main()) [ 794.708894] env[73898]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 794.708894] env[73898]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 794.708894] env[73898]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 794.708894] env[73898]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 794.708894] env[73898]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 794.708894] env[73898]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 794.708894] env[73898]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 794.708894] env[73898]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 794.708894] env[73898]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 794.708894] env[73898]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 794.708894] env[73898]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 794.708894] env[73898]: ERROR neutron response = self.get(path=path, params=params) [ 794.709570] env[73898]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 794.709570] env[73898]: ERROR neutron raise RuntimeError(msg, last_err) [ 794.709570] env[73898]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 794.709570] env[73898]: ERROR neutron [ 797.345874] env[74182]: INFO neutron.common.config [-] Logging enabled! [ 797.346201] env[74182]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 797.346490] env[74182]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74182) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 797.346716] env[74182]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 797.346811] env[74182]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 797.347653] env[74182]: WARNING oslo_policy.policy [None req-a04c6f21-a14c-42bd-8b01-6da31bccf237 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 797.347872] env[74182]: WARNING oslo_policy.policy [None req-a04c6f21-a14c-42bd-8b01-6da31bccf237 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 797.349099] env[74182]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-a04c6f21-a14c-42bd-8b01-6da31bccf237 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 797.362774] env[74182]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a04c6f21-a14c-42bd-8b01-6da31bccf237 None None] Session token - acquiring [ 797.363578] env[74182]: DEBUG networking_nsxv3.common.locking [None req-a04c6f21-a14c-42bd-8b01-6da31bccf237 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74182) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 797.363817] env[74182]: DEBUG oslo_concurrency.lockutils [None req-a04c6f21-a14c-42bd-8b01-6da31bccf237 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74182) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 797.363954] env[74182]: DEBUG oslo_concurrency.lockutils [None req-a04c6f21-a14c-42bd-8b01-6da31bccf237 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74182) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 797.364337] env[74182]: DEBUG oslo_concurrency.lockutils [None req-a04c6f21-a14c-42bd-8b01-6da31bccf237 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74182) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 797.373295] env[74182]: DEBUG oslo_concurrency.lockutils [None req-a04c6f21-a14c-42bd-8b01-6da31bccf237 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74182) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 797.373550] env[74182]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a04c6f21-a14c-42bd-8b01-6da31bccf237 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 797.373696] env[74182]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a04c6f21-a14c-42bd-8b01-6da31bccf237 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74182) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 799.380334] env[74182]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a04c6f21-a14c-42bd-8b01-6da31bccf237 None None] Session token - acquiring [ 799.380720] env[74182]: DEBUG networking_nsxv3.common.locking [None req-a04c6f21-a14c-42bd-8b01-6da31bccf237 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74182) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 799.380843] env[74182]: DEBUG oslo_concurrency.lockutils [None req-a04c6f21-a14c-42bd-8b01-6da31bccf237 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74182) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 799.380959] env[74182]: DEBUG oslo_concurrency.lockutils [None req-a04c6f21-a14c-42bd-8b01-6da31bccf237 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74182) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 799.381229] env[74182]: DEBUG oslo_concurrency.lockutils [None req-a04c6f21-a14c-42bd-8b01-6da31bccf237 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74182) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 799.390318] env[74182]: DEBUG oslo_concurrency.lockutils [None req-a04c6f21-a14c-42bd-8b01-6da31bccf237 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74182) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 799.390562] env[74182]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a04c6f21-a14c-42bd-8b01-6da31bccf237 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 799.390703] env[74182]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a04c6f21-a14c-42bd-8b01-6da31bccf237 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74182) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 801.393505] env[74182]: CRITICAL neutron [None req-a04c6f21-a14c-42bd-8b01-6da31bccf237 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 801.393505] env[74182]: ERROR neutron Traceback (most recent call last): [ 801.393505] env[74182]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 801.393505] env[74182]: ERROR neutron sys.exit(main()) [ 801.393505] env[74182]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 801.393505] env[74182]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 801.393505] env[74182]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 801.393505] env[74182]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 801.393505] env[74182]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 801.393505] env[74182]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 801.393505] env[74182]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 801.393505] env[74182]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 801.393505] env[74182]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 801.393505] env[74182]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 801.393505] env[74182]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 801.393505] env[74182]: ERROR neutron response = self.get(path=path, params=params) [ 801.394195] env[74182]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 801.394195] env[74182]: ERROR neutron raise RuntimeError(msg, last_err) [ 801.394195] env[74182]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 801.394195] env[74182]: ERROR neutron [ 803.896030] env[74280]: INFO neutron.common.config [-] Logging enabled! [ 803.896411] env[74280]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 803.896555] env[74280]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74280) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 803.896786] env[74280]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 803.896881] env[74280]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 803.897710] env[74280]: WARNING oslo_policy.policy [None req-363d826f-3460-4f0e-9749-2514e52a4e5a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 803.897933] env[74280]: WARNING oslo_policy.policy [None req-363d826f-3460-4f0e-9749-2514e52a4e5a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 803.899168] env[74280]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-363d826f-3460-4f0e-9749-2514e52a4e5a None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 803.913209] env[74280]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-363d826f-3460-4f0e-9749-2514e52a4e5a None None] Session token - acquiring [ 803.914002] env[74280]: DEBUG networking_nsxv3.common.locking [None req-363d826f-3460-4f0e-9749-2514e52a4e5a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74280) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 803.914200] env[74280]: DEBUG oslo_concurrency.lockutils [None req-363d826f-3460-4f0e-9749-2514e52a4e5a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74280) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 803.914325] env[74280]: DEBUG oslo_concurrency.lockutils [None req-363d826f-3460-4f0e-9749-2514e52a4e5a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74280) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 803.914671] env[74280]: DEBUG oslo_concurrency.lockutils [None req-363d826f-3460-4f0e-9749-2514e52a4e5a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74280) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 803.923527] env[74280]: DEBUG oslo_concurrency.lockutils [None req-363d826f-3460-4f0e-9749-2514e52a4e5a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74280) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 803.923780] env[74280]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-363d826f-3460-4f0e-9749-2514e52a4e5a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 803.923933] env[74280]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-363d826f-3460-4f0e-9749-2514e52a4e5a None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74280) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 805.930013] env[74280]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-363d826f-3460-4f0e-9749-2514e52a4e5a None None] Session token - acquiring [ 805.930473] env[74280]: DEBUG networking_nsxv3.common.locking [None req-363d826f-3460-4f0e-9749-2514e52a4e5a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74280) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 805.930672] env[74280]: DEBUG oslo_concurrency.lockutils [None req-363d826f-3460-4f0e-9749-2514e52a4e5a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74280) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 805.930792] env[74280]: DEBUG oslo_concurrency.lockutils [None req-363d826f-3460-4f0e-9749-2514e52a4e5a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74280) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 805.931066] env[74280]: DEBUG oslo_concurrency.lockutils [None req-363d826f-3460-4f0e-9749-2514e52a4e5a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74280) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 805.941092] env[74280]: DEBUG oslo_concurrency.lockutils [None req-363d826f-3460-4f0e-9749-2514e52a4e5a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74280) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 805.941439] env[74280]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-363d826f-3460-4f0e-9749-2514e52a4e5a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 805.941511] env[74280]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-363d826f-3460-4f0e-9749-2514e52a4e5a None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74280) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 807.944445] env[74280]: CRITICAL neutron [None req-363d826f-3460-4f0e-9749-2514e52a4e5a None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 807.944445] env[74280]: ERROR neutron Traceback (most recent call last): [ 807.944445] env[74280]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 807.944445] env[74280]: ERROR neutron sys.exit(main()) [ 807.944445] env[74280]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 807.944445] env[74280]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 807.944445] env[74280]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 807.944445] env[74280]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 807.944445] env[74280]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 807.944445] env[74280]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 807.944445] env[74280]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 807.944445] env[74280]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 807.944445] env[74280]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 807.944445] env[74280]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 807.944445] env[74280]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 807.944445] env[74280]: ERROR neutron response = self.get(path=path, params=params) [ 807.945299] env[74280]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 807.945299] env[74280]: ERROR neutron raise RuntimeError(msg, last_err) [ 807.945299] env[74280]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 807.945299] env[74280]: ERROR neutron [ 810.643478] env[74333]: INFO neutron.common.config [-] Logging enabled! [ 810.643837] env[74333]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 810.644043] env[74333]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74333) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 810.644296] env[74333]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 810.644393] env[74333]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 810.645311] env[74333]: WARNING oslo_policy.policy [None req-105cac97-96f1-4b9e-94a4-27e199a8692a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 810.645505] env[74333]: WARNING oslo_policy.policy [None req-105cac97-96f1-4b9e-94a4-27e199a8692a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 810.646704] env[74333]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-105cac97-96f1-4b9e-94a4-27e199a8692a None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 810.661010] env[74333]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-105cac97-96f1-4b9e-94a4-27e199a8692a None None] Session token - acquiring [ 810.661970] env[74333]: DEBUG networking_nsxv3.common.locking [None req-105cac97-96f1-4b9e-94a4-27e199a8692a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74333) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 810.662189] env[74333]: DEBUG oslo_concurrency.lockutils [None req-105cac97-96f1-4b9e-94a4-27e199a8692a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74333) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 810.662321] env[74333]: DEBUG oslo_concurrency.lockutils [None req-105cac97-96f1-4b9e-94a4-27e199a8692a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74333) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 810.662691] env[74333]: DEBUG oslo_concurrency.lockutils [None req-105cac97-96f1-4b9e-94a4-27e199a8692a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74333) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 810.671801] env[74333]: DEBUG oslo_concurrency.lockutils [None req-105cac97-96f1-4b9e-94a4-27e199a8692a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74333) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 810.672044] env[74333]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-105cac97-96f1-4b9e-94a4-27e199a8692a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 810.672199] env[74333]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-105cac97-96f1-4b9e-94a4-27e199a8692a None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74333) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 812.680679] env[74333]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-105cac97-96f1-4b9e-94a4-27e199a8692a None None] Session token - acquiring [ 812.681000] env[74333]: DEBUG networking_nsxv3.common.locking [None req-105cac97-96f1-4b9e-94a4-27e199a8692a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74333) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 812.681196] env[74333]: DEBUG oslo_concurrency.lockutils [None req-105cac97-96f1-4b9e-94a4-27e199a8692a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74333) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 812.681324] env[74333]: DEBUG oslo_concurrency.lockutils [None req-105cac97-96f1-4b9e-94a4-27e199a8692a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74333) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 812.681602] env[74333]: DEBUG oslo_concurrency.lockutils [None req-105cac97-96f1-4b9e-94a4-27e199a8692a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74333) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 812.690954] env[74333]: DEBUG oslo_concurrency.lockutils [None req-105cac97-96f1-4b9e-94a4-27e199a8692a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74333) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 812.691192] env[74333]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-105cac97-96f1-4b9e-94a4-27e199a8692a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 812.691333] env[74333]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-105cac97-96f1-4b9e-94a4-27e199a8692a None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74333) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 814.692894] env[74333]: CRITICAL neutron [None req-105cac97-96f1-4b9e-94a4-27e199a8692a None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 814.692894] env[74333]: ERROR neutron Traceback (most recent call last): [ 814.692894] env[74333]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 814.692894] env[74333]: ERROR neutron sys.exit(main()) [ 814.692894] env[74333]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 814.692894] env[74333]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 814.692894] env[74333]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 814.692894] env[74333]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 814.692894] env[74333]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 814.692894] env[74333]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 814.692894] env[74333]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 814.692894] env[74333]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 814.692894] env[74333]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 814.692894] env[74333]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 814.692894] env[74333]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 814.692894] env[74333]: ERROR neutron response = self.get(path=path, params=params) [ 814.693553] env[74333]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 814.693553] env[74333]: ERROR neutron raise RuntimeError(msg, last_err) [ 814.693553] env[74333]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 814.693553] env[74333]: ERROR neutron [ 817.511664] env[74404]: INFO neutron.common.config [-] Logging enabled! [ 817.512195] env[74404]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 817.512195] env[74404]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74404) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 817.512451] env[74404]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 817.512546] env[74404]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 817.513438] env[74404]: WARNING oslo_policy.policy [None req-a2c3f0fe-d23e-418e-9eae-a3efa2597d71 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 817.513651] env[74404]: WARNING oslo_policy.policy [None req-a2c3f0fe-d23e-418e-9eae-a3efa2597d71 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 817.514927] env[74404]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-a2c3f0fe-d23e-418e-9eae-a3efa2597d71 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 817.529877] env[74404]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a2c3f0fe-d23e-418e-9eae-a3efa2597d71 None None] Session token - acquiring [ 817.530729] env[74404]: DEBUG networking_nsxv3.common.locking [None req-a2c3f0fe-d23e-418e-9eae-a3efa2597d71 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74404) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 817.530943] env[74404]: DEBUG oslo_concurrency.lockutils [None req-a2c3f0fe-d23e-418e-9eae-a3efa2597d71 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74404) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 817.531070] env[74404]: DEBUG oslo_concurrency.lockutils [None req-a2c3f0fe-d23e-418e-9eae-a3efa2597d71 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74404) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 817.531441] env[74404]: DEBUG oslo_concurrency.lockutils [None req-a2c3f0fe-d23e-418e-9eae-a3efa2597d71 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74404) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 817.540941] env[74404]: DEBUG oslo_concurrency.lockutils [None req-a2c3f0fe-d23e-418e-9eae-a3efa2597d71 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74404) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 817.541199] env[74404]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a2c3f0fe-d23e-418e-9eae-a3efa2597d71 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 817.541344] env[74404]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a2c3f0fe-d23e-418e-9eae-a3efa2597d71 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74404) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 819.548558] env[74404]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a2c3f0fe-d23e-418e-9eae-a3efa2597d71 None None] Session token - acquiring [ 819.548558] env[74404]: DEBUG networking_nsxv3.common.locking [None req-a2c3f0fe-d23e-418e-9eae-a3efa2597d71 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74404) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 819.548558] env[74404]: DEBUG oslo_concurrency.lockutils [None req-a2c3f0fe-d23e-418e-9eae-a3efa2597d71 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74404) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 819.548924] env[74404]: DEBUG oslo_concurrency.lockutils [None req-a2c3f0fe-d23e-418e-9eae-a3efa2597d71 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74404) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 819.548924] env[74404]: DEBUG oslo_concurrency.lockutils [None req-a2c3f0fe-d23e-418e-9eae-a3efa2597d71 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74404) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 819.557633] env[74404]: DEBUG oslo_concurrency.lockutils [None req-a2c3f0fe-d23e-418e-9eae-a3efa2597d71 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74404) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 819.558009] env[74404]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a2c3f0fe-d23e-418e-9eae-a3efa2597d71 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 819.558200] env[74404]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a2c3f0fe-d23e-418e-9eae-a3efa2597d71 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74404) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 821.560880] env[74404]: CRITICAL neutron [None req-a2c3f0fe-d23e-418e-9eae-a3efa2597d71 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 821.560880] env[74404]: ERROR neutron Traceback (most recent call last): [ 821.560880] env[74404]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 821.560880] env[74404]: ERROR neutron sys.exit(main()) [ 821.560880] env[74404]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 821.560880] env[74404]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 821.560880] env[74404]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 821.560880] env[74404]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 821.560880] env[74404]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 821.560880] env[74404]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 821.560880] env[74404]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 821.560880] env[74404]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 821.560880] env[74404]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 821.560880] env[74404]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 821.560880] env[74404]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 821.560880] env[74404]: ERROR neutron response = self.get(path=path, params=params) [ 821.561572] env[74404]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 821.561572] env[74404]: ERROR neutron raise RuntimeError(msg, last_err) [ 821.561572] env[74404]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 821.561572] env[74404]: ERROR neutron [ 824.169544] env[74453]: INFO neutron.common.config [-] Logging enabled! [ 824.169988] env[74453]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 824.170119] env[74453]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74453) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 824.170359] env[74453]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 824.170455] env[74453]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 824.171319] env[74453]: WARNING oslo_policy.policy [None req-c7bd4c26-6125-47e0-85e9-a4b7a7d0f0ac None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 824.171563] env[74453]: WARNING oslo_policy.policy [None req-c7bd4c26-6125-47e0-85e9-a4b7a7d0f0ac None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 824.172876] env[74453]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-c7bd4c26-6125-47e0-85e9-a4b7a7d0f0ac None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 824.187177] env[74453]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c7bd4c26-6125-47e0-85e9-a4b7a7d0f0ac None None] Session token - acquiring [ 824.188061] env[74453]: DEBUG networking_nsxv3.common.locking [None req-c7bd4c26-6125-47e0-85e9-a4b7a7d0f0ac None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74453) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 824.188243] env[74453]: DEBUG oslo_concurrency.lockutils [None req-c7bd4c26-6125-47e0-85e9-a4b7a7d0f0ac None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74453) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 824.188371] env[74453]: DEBUG oslo_concurrency.lockutils [None req-c7bd4c26-6125-47e0-85e9-a4b7a7d0f0ac None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74453) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 824.188776] env[74453]: DEBUG oslo_concurrency.lockutils [None req-c7bd4c26-6125-47e0-85e9-a4b7a7d0f0ac None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74453) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 824.197892] env[74453]: DEBUG oslo_concurrency.lockutils [None req-c7bd4c26-6125-47e0-85e9-a4b7a7d0f0ac None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74453) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 824.198173] env[74453]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c7bd4c26-6125-47e0-85e9-a4b7a7d0f0ac None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 824.198280] env[74453]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c7bd4c26-6125-47e0-85e9-a4b7a7d0f0ac None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74453) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 826.204281] env[74453]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c7bd4c26-6125-47e0-85e9-a4b7a7d0f0ac None None] Session token - acquiring [ 826.204584] env[74453]: DEBUG networking_nsxv3.common.locking [None req-c7bd4c26-6125-47e0-85e9-a4b7a7d0f0ac None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74453) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 826.204807] env[74453]: DEBUG oslo_concurrency.lockutils [None req-c7bd4c26-6125-47e0-85e9-a4b7a7d0f0ac None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74453) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 826.204903] env[74453]: DEBUG oslo_concurrency.lockutils [None req-c7bd4c26-6125-47e0-85e9-a4b7a7d0f0ac None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74453) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 826.205174] env[74453]: DEBUG oslo_concurrency.lockutils [None req-c7bd4c26-6125-47e0-85e9-a4b7a7d0f0ac None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74453) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 826.215069] env[74453]: DEBUG oslo_concurrency.lockutils [None req-c7bd4c26-6125-47e0-85e9-a4b7a7d0f0ac None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74453) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 826.215327] env[74453]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c7bd4c26-6125-47e0-85e9-a4b7a7d0f0ac None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 826.215463] env[74453]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c7bd4c26-6125-47e0-85e9-a4b7a7d0f0ac None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74453) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 828.217782] env[74453]: CRITICAL neutron [None req-c7bd4c26-6125-47e0-85e9-a4b7a7d0f0ac None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 828.217782] env[74453]: ERROR neutron Traceback (most recent call last): [ 828.217782] env[74453]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 828.217782] env[74453]: ERROR neutron sys.exit(main()) [ 828.217782] env[74453]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 828.217782] env[74453]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 828.217782] env[74453]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 828.217782] env[74453]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 828.217782] env[74453]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 828.217782] env[74453]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 828.217782] env[74453]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 828.217782] env[74453]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 828.217782] env[74453]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 828.217782] env[74453]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 828.217782] env[74453]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 828.217782] env[74453]: ERROR neutron response = self.get(path=path, params=params) [ 828.218529] env[74453]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 828.218529] env[74453]: ERROR neutron raise RuntimeError(msg, last_err) [ 828.218529] env[74453]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 828.218529] env[74453]: ERROR neutron [ 830.927703] env[74524]: INFO neutron.common.config [-] Logging enabled! [ 830.928037] env[74524]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 830.928235] env[74524]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74524) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 830.928462] env[74524]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 830.928553] env[74524]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 830.929396] env[74524]: WARNING oslo_policy.policy [None req-c57f6166-f116-4445-91cc-f0282dee3a33 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 830.929620] env[74524]: WARNING oslo_policy.policy [None req-c57f6166-f116-4445-91cc-f0282dee3a33 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 830.930823] env[74524]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-c57f6166-f116-4445-91cc-f0282dee3a33 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 830.946219] env[74524]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c57f6166-f116-4445-91cc-f0282dee3a33 None None] Session token - acquiring [ 830.947231] env[74524]: DEBUG networking_nsxv3.common.locking [None req-c57f6166-f116-4445-91cc-f0282dee3a33 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74524) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 830.947587] env[74524]: DEBUG oslo_concurrency.lockutils [None req-c57f6166-f116-4445-91cc-f0282dee3a33 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74524) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 830.947840] env[74524]: DEBUG oslo_concurrency.lockutils [None req-c57f6166-f116-4445-91cc-f0282dee3a33 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74524) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 830.948415] env[74524]: DEBUG oslo_concurrency.lockutils [None req-c57f6166-f116-4445-91cc-f0282dee3a33 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74524) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 830.961340] env[74524]: DEBUG oslo_concurrency.lockutils [None req-c57f6166-f116-4445-91cc-f0282dee3a33 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74524) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 830.961768] env[74524]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c57f6166-f116-4445-91cc-f0282dee3a33 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 830.962040] env[74524]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c57f6166-f116-4445-91cc-f0282dee3a33 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74524) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 832.974281] env[74524]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c57f6166-f116-4445-91cc-f0282dee3a33 None None] Session token - acquiring [ 832.974281] env[74524]: DEBUG networking_nsxv3.common.locking [None req-c57f6166-f116-4445-91cc-f0282dee3a33 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74524) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 832.974281] env[74524]: DEBUG oslo_concurrency.lockutils [None req-c57f6166-f116-4445-91cc-f0282dee3a33 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74524) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 832.974679] env[74524]: DEBUG oslo_concurrency.lockutils [None req-c57f6166-f116-4445-91cc-f0282dee3a33 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74524) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 832.974679] env[74524]: DEBUG oslo_concurrency.lockutils [None req-c57f6166-f116-4445-91cc-f0282dee3a33 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74524) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 832.983828] env[74524]: DEBUG oslo_concurrency.lockutils [None req-c57f6166-f116-4445-91cc-f0282dee3a33 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74524) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 832.984158] env[74524]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c57f6166-f116-4445-91cc-f0282dee3a33 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 832.984378] env[74524]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c57f6166-f116-4445-91cc-f0282dee3a33 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74524) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 834.987276] env[74524]: CRITICAL neutron [None req-c57f6166-f116-4445-91cc-f0282dee3a33 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 834.987276] env[74524]: ERROR neutron Traceback (most recent call last): [ 834.987276] env[74524]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 834.987276] env[74524]: ERROR neutron sys.exit(main()) [ 834.987276] env[74524]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 834.987276] env[74524]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 834.987276] env[74524]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 834.987276] env[74524]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 834.987276] env[74524]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 834.987276] env[74524]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 834.987276] env[74524]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 834.987276] env[74524]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 834.987276] env[74524]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 834.987276] env[74524]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 834.987276] env[74524]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 834.987276] env[74524]: ERROR neutron response = self.get(path=path, params=params) [ 834.988098] env[74524]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 834.988098] env[74524]: ERROR neutron raise RuntimeError(msg, last_err) [ 834.988098] env[74524]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 834.988098] env[74524]: ERROR neutron [ 837.738427] env[74656]: INFO neutron.common.config [-] Logging enabled! [ 837.738659] env[74656]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 837.738992] env[74656]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74656) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 837.739172] env[74656]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 837.739264] env[74656]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 837.740221] env[74656]: WARNING oslo_policy.policy [None req-43f2756d-98f2-4dcd-a660-099903de6af6 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 837.740469] env[74656]: WARNING oslo_policy.policy [None req-43f2756d-98f2-4dcd-a660-099903de6af6 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 837.741726] env[74656]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-43f2756d-98f2-4dcd-a660-099903de6af6 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 837.756236] env[74656]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-43f2756d-98f2-4dcd-a660-099903de6af6 None None] Session token - acquiring [ 837.757069] env[74656]: DEBUG networking_nsxv3.common.locking [None req-43f2756d-98f2-4dcd-a660-099903de6af6 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74656) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 837.757277] env[74656]: DEBUG oslo_concurrency.lockutils [None req-43f2756d-98f2-4dcd-a660-099903de6af6 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74656) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 837.757386] env[74656]: DEBUG oslo_concurrency.lockutils [None req-43f2756d-98f2-4dcd-a660-099903de6af6 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74656) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 837.757747] env[74656]: DEBUG oslo_concurrency.lockutils [None req-43f2756d-98f2-4dcd-a660-099903de6af6 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74656) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 837.768078] env[74656]: DEBUG oslo_concurrency.lockutils [None req-43f2756d-98f2-4dcd-a660-099903de6af6 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74656) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 837.768337] env[74656]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-43f2756d-98f2-4dcd-a660-099903de6af6 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 837.768479] env[74656]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-43f2756d-98f2-4dcd-a660-099903de6af6 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74656) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 839.776395] env[74656]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-43f2756d-98f2-4dcd-a660-099903de6af6 None None] Session token - acquiring [ 839.776736] env[74656]: DEBUG networking_nsxv3.common.locking [None req-43f2756d-98f2-4dcd-a660-099903de6af6 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74656) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 839.776892] env[74656]: DEBUG oslo_concurrency.lockutils [None req-43f2756d-98f2-4dcd-a660-099903de6af6 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74656) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 839.777054] env[74656]: DEBUG oslo_concurrency.lockutils [None req-43f2756d-98f2-4dcd-a660-099903de6af6 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74656) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 839.777458] env[74656]: DEBUG oslo_concurrency.lockutils [None req-43f2756d-98f2-4dcd-a660-099903de6af6 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74656) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 839.786635] env[74656]: DEBUG oslo_concurrency.lockutils [None req-43f2756d-98f2-4dcd-a660-099903de6af6 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74656) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 839.786862] env[74656]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-43f2756d-98f2-4dcd-a660-099903de6af6 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 839.787082] env[74656]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-43f2756d-98f2-4dcd-a660-099903de6af6 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74656) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 841.789195] env[74656]: CRITICAL neutron [None req-43f2756d-98f2-4dcd-a660-099903de6af6 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 841.789195] env[74656]: ERROR neutron Traceback (most recent call last): [ 841.789195] env[74656]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 841.789195] env[74656]: ERROR neutron sys.exit(main()) [ 841.789195] env[74656]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 841.789195] env[74656]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 841.789195] env[74656]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 841.789195] env[74656]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 841.789195] env[74656]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 841.789195] env[74656]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 841.789195] env[74656]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 841.789195] env[74656]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 841.789195] env[74656]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 841.789195] env[74656]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 841.789195] env[74656]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 841.789195] env[74656]: ERROR neutron response = self.get(path=path, params=params) [ 841.789932] env[74656]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 841.789932] env[74656]: ERROR neutron raise RuntimeError(msg, last_err) [ 841.789932] env[74656]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 841.789932] env[74656]: ERROR neutron [ 844.362649] env[74869]: INFO neutron.common.config [-] Logging enabled! [ 844.362992] env[74869]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 844.363145] env[74869]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74869) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 844.363393] env[74869]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 844.363488] env[74869]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 844.364406] env[74869]: WARNING oslo_policy.policy [None req-8476a3d0-f51f-4244-8f3e-e13aa61bdb9a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 844.364631] env[74869]: WARNING oslo_policy.policy [None req-8476a3d0-f51f-4244-8f3e-e13aa61bdb9a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 844.366195] env[74869]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-8476a3d0-f51f-4244-8f3e-e13aa61bdb9a None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 844.380328] env[74869]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8476a3d0-f51f-4244-8f3e-e13aa61bdb9a None None] Session token - acquiring [ 844.381107] env[74869]: DEBUG networking_nsxv3.common.locking [None req-8476a3d0-f51f-4244-8f3e-e13aa61bdb9a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74869) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 844.381312] env[74869]: DEBUG oslo_concurrency.lockutils [None req-8476a3d0-f51f-4244-8f3e-e13aa61bdb9a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74869) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 844.381439] env[74869]: DEBUG oslo_concurrency.lockutils [None req-8476a3d0-f51f-4244-8f3e-e13aa61bdb9a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74869) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 844.381809] env[74869]: DEBUG oslo_concurrency.lockutils [None req-8476a3d0-f51f-4244-8f3e-e13aa61bdb9a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74869) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 844.390634] env[74869]: DEBUG oslo_concurrency.lockutils [None req-8476a3d0-f51f-4244-8f3e-e13aa61bdb9a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74869) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 844.390892] env[74869]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8476a3d0-f51f-4244-8f3e-e13aa61bdb9a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 844.391041] env[74869]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8476a3d0-f51f-4244-8f3e-e13aa61bdb9a None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74869) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 846.399539] env[74869]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8476a3d0-f51f-4244-8f3e-e13aa61bdb9a None None] Session token - acquiring [ 846.399539] env[74869]: DEBUG networking_nsxv3.common.locking [None req-8476a3d0-f51f-4244-8f3e-e13aa61bdb9a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74869) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 846.399539] env[74869]: DEBUG oslo_concurrency.lockutils [None req-8476a3d0-f51f-4244-8f3e-e13aa61bdb9a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74869) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 846.400057] env[74869]: DEBUG oslo_concurrency.lockutils [None req-8476a3d0-f51f-4244-8f3e-e13aa61bdb9a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74869) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 846.400057] env[74869]: DEBUG oslo_concurrency.lockutils [None req-8476a3d0-f51f-4244-8f3e-e13aa61bdb9a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74869) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 846.408437] env[74869]: DEBUG oslo_concurrency.lockutils [None req-8476a3d0-f51f-4244-8f3e-e13aa61bdb9a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74869) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 846.408437] env[74869]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8476a3d0-f51f-4244-8f3e-e13aa61bdb9a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 846.408437] env[74869]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8476a3d0-f51f-4244-8f3e-e13aa61bdb9a None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74869) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 848.411404] env[74869]: CRITICAL neutron [None req-8476a3d0-f51f-4244-8f3e-e13aa61bdb9a None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 848.411404] env[74869]: ERROR neutron Traceback (most recent call last): [ 848.411404] env[74869]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 848.411404] env[74869]: ERROR neutron sys.exit(main()) [ 848.411404] env[74869]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 848.411404] env[74869]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 848.411404] env[74869]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 848.411404] env[74869]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 848.411404] env[74869]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 848.411404] env[74869]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 848.411404] env[74869]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 848.411404] env[74869]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 848.411404] env[74869]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 848.411404] env[74869]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 848.411404] env[74869]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 848.411404] env[74869]: ERROR neutron response = self.get(path=path, params=params) [ 848.412131] env[74869]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 848.412131] env[74869]: ERROR neutron raise RuntimeError(msg, last_err) [ 848.412131] env[74869]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 848.412131] env[74869]: ERROR neutron [ 850.873120] env[74994]: INFO neutron.common.config [-] Logging enabled! [ 850.873485] env[74994]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 850.873624] env[74994]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74994) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 850.873855] env[74994]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 850.873951] env[74994]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 850.874779] env[74994]: WARNING oslo_policy.policy [None req-215c51c9-2f94-44c9-b1bd-457453495a5b None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 850.875004] env[74994]: WARNING oslo_policy.policy [None req-215c51c9-2f94-44c9-b1bd-457453495a5b None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 850.876197] env[74994]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-215c51c9-2f94-44c9-b1bd-457453495a5b None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 850.889802] env[74994]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-215c51c9-2f94-44c9-b1bd-457453495a5b None None] Session token - acquiring [ 850.890529] env[74994]: DEBUG networking_nsxv3.common.locking [None req-215c51c9-2f94-44c9-b1bd-457453495a5b None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74994) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 850.890725] env[74994]: DEBUG oslo_concurrency.lockutils [None req-215c51c9-2f94-44c9-b1bd-457453495a5b None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74994) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 850.890852] env[74994]: DEBUG oslo_concurrency.lockutils [None req-215c51c9-2f94-44c9-b1bd-457453495a5b None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74994) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 850.891210] env[74994]: DEBUG oslo_concurrency.lockutils [None req-215c51c9-2f94-44c9-b1bd-457453495a5b None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74994) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 850.899926] env[74994]: DEBUG oslo_concurrency.lockutils [None req-215c51c9-2f94-44c9-b1bd-457453495a5b None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74994) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 850.900186] env[74994]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-215c51c9-2f94-44c9-b1bd-457453495a5b None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 850.900334] env[74994]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-215c51c9-2f94-44c9-b1bd-457453495a5b None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74994) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 852.915226] env[74994]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-215c51c9-2f94-44c9-b1bd-457453495a5b None None] Session token - acquiring [ 852.915564] env[74994]: DEBUG networking_nsxv3.common.locking [None req-215c51c9-2f94-44c9-b1bd-457453495a5b None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74994) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 852.915752] env[74994]: DEBUG oslo_concurrency.lockutils [None req-215c51c9-2f94-44c9-b1bd-457453495a5b None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74994) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 852.915878] env[74994]: DEBUG oslo_concurrency.lockutils [None req-215c51c9-2f94-44c9-b1bd-457453495a5b None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74994) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 852.916213] env[74994]: DEBUG oslo_concurrency.lockutils [None req-215c51c9-2f94-44c9-b1bd-457453495a5b None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74994) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 852.925235] env[74994]: DEBUG oslo_concurrency.lockutils [None req-215c51c9-2f94-44c9-b1bd-457453495a5b None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74994) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 852.925486] env[74994]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-215c51c9-2f94-44c9-b1bd-457453495a5b None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 852.925649] env[74994]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-215c51c9-2f94-44c9-b1bd-457453495a5b None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74994) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 854.928560] env[74994]: CRITICAL neutron [None req-215c51c9-2f94-44c9-b1bd-457453495a5b None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 854.928560] env[74994]: ERROR neutron Traceback (most recent call last): [ 854.928560] env[74994]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 854.928560] env[74994]: ERROR neutron sys.exit(main()) [ 854.928560] env[74994]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 854.928560] env[74994]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 854.928560] env[74994]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 854.928560] env[74994]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 854.928560] env[74994]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 854.928560] env[74994]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 854.928560] env[74994]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 854.928560] env[74994]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 854.928560] env[74994]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 854.928560] env[74994]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 854.928560] env[74994]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 854.928560] env[74994]: ERROR neutron response = self.get(path=path, params=params) [ 854.929245] env[74994]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 854.929245] env[74994]: ERROR neutron raise RuntimeError(msg, last_err) [ 854.929245] env[74994]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 854.929245] env[74994]: ERROR neutron [ 857.339802] env[75107]: INFO neutron.common.config [-] Logging enabled! [ 857.340270] env[75107]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 857.340355] env[75107]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=75107) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 857.340600] env[75107]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 857.340695] env[75107]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 857.341536] env[75107]: WARNING oslo_policy.policy [None req-59bf12c5-2382-47d1-9c27-919da432b934 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 857.341783] env[75107]: WARNING oslo_policy.policy [None req-59bf12c5-2382-47d1-9c27-919da432b934 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 857.343015] env[75107]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-59bf12c5-2382-47d1-9c27-919da432b934 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 857.357697] env[75107]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-59bf12c5-2382-47d1-9c27-919da432b934 None None] Session token - acquiring [ 857.358574] env[75107]: DEBUG networking_nsxv3.common.locking [None req-59bf12c5-2382-47d1-9c27-919da432b934 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75107) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 857.358781] env[75107]: DEBUG oslo_concurrency.lockutils [None req-59bf12c5-2382-47d1-9c27-919da432b934 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75107) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 857.358912] env[75107]: DEBUG oslo_concurrency.lockutils [None req-59bf12c5-2382-47d1-9c27-919da432b934 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75107) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 857.359289] env[75107]: DEBUG oslo_concurrency.lockutils [None req-59bf12c5-2382-47d1-9c27-919da432b934 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75107) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 857.369955] env[75107]: DEBUG oslo_concurrency.lockutils [None req-59bf12c5-2382-47d1-9c27-919da432b934 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75107) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 857.370166] env[75107]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-59bf12c5-2382-47d1-9c27-919da432b934 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 857.370314] env[75107]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-59bf12c5-2382-47d1-9c27-919da432b934 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=75107) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 859.378413] env[75107]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-59bf12c5-2382-47d1-9c27-919da432b934 None None] Session token - acquiring [ 859.378413] env[75107]: DEBUG networking_nsxv3.common.locking [None req-59bf12c5-2382-47d1-9c27-919da432b934 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75107) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 859.378413] env[75107]: DEBUG oslo_concurrency.lockutils [None req-59bf12c5-2382-47d1-9c27-919da432b934 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75107) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 859.378866] env[75107]: DEBUG oslo_concurrency.lockutils [None req-59bf12c5-2382-47d1-9c27-919da432b934 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75107) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 859.378866] env[75107]: DEBUG oslo_concurrency.lockutils [None req-59bf12c5-2382-47d1-9c27-919da432b934 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75107) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 859.387901] env[75107]: DEBUG oslo_concurrency.lockutils [None req-59bf12c5-2382-47d1-9c27-919da432b934 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75107) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 859.387901] env[75107]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-59bf12c5-2382-47d1-9c27-919da432b934 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 859.387901] env[75107]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-59bf12c5-2382-47d1-9c27-919da432b934 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=75107) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 861.388932] env[75107]: CRITICAL neutron [None req-59bf12c5-2382-47d1-9c27-919da432b934 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 861.388932] env[75107]: ERROR neutron Traceback (most recent call last): [ 861.388932] env[75107]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 861.388932] env[75107]: ERROR neutron sys.exit(main()) [ 861.388932] env[75107]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 861.388932] env[75107]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 861.388932] env[75107]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 861.388932] env[75107]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 861.388932] env[75107]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 861.388932] env[75107]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 861.388932] env[75107]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 861.388932] env[75107]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 861.388932] env[75107]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 861.388932] env[75107]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 861.388932] env[75107]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 861.388932] env[75107]: ERROR neutron response = self.get(path=path, params=params) [ 861.389704] env[75107]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 861.389704] env[75107]: ERROR neutron raise RuntimeError(msg, last_err) [ 861.389704] env[75107]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 861.389704] env[75107]: ERROR neutron [ 863.858124] env[75136]: INFO neutron.common.config [-] Logging enabled! [ 863.858499] env[75136]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 863.858678] env[75136]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=75136) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 863.858905] env[75136]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 863.859002] env[75136]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 863.859844] env[75136]: WARNING oslo_policy.policy [None req-23957af1-faf6-46a9-938e-5550e0dd99c9 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 863.860082] env[75136]: WARNING oslo_policy.policy [None req-23957af1-faf6-46a9-938e-5550e0dd99c9 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 863.861298] env[75136]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-23957af1-faf6-46a9-938e-5550e0dd99c9 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 863.876252] env[75136]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-23957af1-faf6-46a9-938e-5550e0dd99c9 None None] Session token - acquiring [ 863.877300] env[75136]: DEBUG networking_nsxv3.common.locking [None req-23957af1-faf6-46a9-938e-5550e0dd99c9 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75136) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 863.877509] env[75136]: DEBUG oslo_concurrency.lockutils [None req-23957af1-faf6-46a9-938e-5550e0dd99c9 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75136) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 863.877799] env[75136]: DEBUG oslo_concurrency.lockutils [None req-23957af1-faf6-46a9-938e-5550e0dd99c9 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75136) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 863.878198] env[75136]: DEBUG oslo_concurrency.lockutils [None req-23957af1-faf6-46a9-938e-5550e0dd99c9 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75136) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 863.888762] env[75136]: DEBUG oslo_concurrency.lockutils [None req-23957af1-faf6-46a9-938e-5550e0dd99c9 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75136) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 863.888762] env[75136]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-23957af1-faf6-46a9-938e-5550e0dd99c9 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 863.888762] env[75136]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-23957af1-faf6-46a9-938e-5550e0dd99c9 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=75136) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 865.894957] env[75136]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-23957af1-faf6-46a9-938e-5550e0dd99c9 None None] Session token - acquiring [ 865.895384] env[75136]: DEBUG networking_nsxv3.common.locking [None req-23957af1-faf6-46a9-938e-5550e0dd99c9 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75136) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 865.895591] env[75136]: DEBUG oslo_concurrency.lockutils [None req-23957af1-faf6-46a9-938e-5550e0dd99c9 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75136) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 865.895722] env[75136]: DEBUG oslo_concurrency.lockutils [None req-23957af1-faf6-46a9-938e-5550e0dd99c9 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75136) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 865.896017] env[75136]: DEBUG oslo_concurrency.lockutils [None req-23957af1-faf6-46a9-938e-5550e0dd99c9 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75136) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 865.905405] env[75136]: DEBUG oslo_concurrency.lockutils [None req-23957af1-faf6-46a9-938e-5550e0dd99c9 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75136) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 865.905658] env[75136]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-23957af1-faf6-46a9-938e-5550e0dd99c9 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 865.905800] env[75136]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-23957af1-faf6-46a9-938e-5550e0dd99c9 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=75136) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 867.909005] env[75136]: CRITICAL neutron [None req-23957af1-faf6-46a9-938e-5550e0dd99c9 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 867.909005] env[75136]: ERROR neutron Traceback (most recent call last): [ 867.909005] env[75136]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 867.909005] env[75136]: ERROR neutron sys.exit(main()) [ 867.909005] env[75136]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 867.909005] env[75136]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 867.909005] env[75136]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 867.909005] env[75136]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 867.909005] env[75136]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 867.909005] env[75136]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 867.909005] env[75136]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 867.909005] env[75136]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 867.909005] env[75136]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 867.909005] env[75136]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 867.909005] env[75136]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 867.909005] env[75136]: ERROR neutron response = self.get(path=path, params=params) [ 867.909708] env[75136]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 867.909708] env[75136]: ERROR neutron raise RuntimeError(msg, last_err) [ 867.909708] env[75136]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 867.909708] env[75136]: ERROR neutron [ 870.612276] env[75227]: INFO neutron.common.config [-] Logging enabled! [ 870.612595] env[75227]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 870.612795] env[75227]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=75227) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 870.613027] env[75227]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 870.613225] env[75227]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 870.613970] env[75227]: WARNING oslo_policy.policy [None req-6fe23df6-0b08-431d-980f-aaa45d8d8093 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 870.614215] env[75227]: WARNING oslo_policy.policy [None req-6fe23df6-0b08-431d-980f-aaa45d8d8093 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 870.615459] env[75227]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-6fe23df6-0b08-431d-980f-aaa45d8d8093 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 870.630537] env[75227]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6fe23df6-0b08-431d-980f-aaa45d8d8093 None None] Session token - acquiring [ 870.631623] env[75227]: DEBUG networking_nsxv3.common.locking [None req-6fe23df6-0b08-431d-980f-aaa45d8d8093 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75227) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 870.632056] env[75227]: DEBUG oslo_concurrency.lockutils [None req-6fe23df6-0b08-431d-980f-aaa45d8d8093 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75227) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 870.632221] env[75227]: DEBUG oslo_concurrency.lockutils [None req-6fe23df6-0b08-431d-980f-aaa45d8d8093 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75227) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 870.632610] env[75227]: DEBUG oslo_concurrency.lockutils [None req-6fe23df6-0b08-431d-980f-aaa45d8d8093 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75227) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 870.649644] env[75227]: DEBUG oslo_concurrency.lockutils [None req-6fe23df6-0b08-431d-980f-aaa45d8d8093 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75227) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 870.649908] env[75227]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6fe23df6-0b08-431d-980f-aaa45d8d8093 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 870.650056] env[75227]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6fe23df6-0b08-431d-980f-aaa45d8d8093 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=75227) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 872.657907] env[75227]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6fe23df6-0b08-431d-980f-aaa45d8d8093 None None] Session token - acquiring [ 872.657907] env[75227]: DEBUG networking_nsxv3.common.locking [None req-6fe23df6-0b08-431d-980f-aaa45d8d8093 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75227) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 872.658255] env[75227]: DEBUG oslo_concurrency.lockutils [None req-6fe23df6-0b08-431d-980f-aaa45d8d8093 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75227) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 872.658255] env[75227]: DEBUG oslo_concurrency.lockutils [None req-6fe23df6-0b08-431d-980f-aaa45d8d8093 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75227) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 872.658383] env[75227]: DEBUG oslo_concurrency.lockutils [None req-6fe23df6-0b08-431d-980f-aaa45d8d8093 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75227) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 872.668207] env[75227]: DEBUG oslo_concurrency.lockutils [None req-6fe23df6-0b08-431d-980f-aaa45d8d8093 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75227) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 872.668450] env[75227]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6fe23df6-0b08-431d-980f-aaa45d8d8093 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 872.668590] env[75227]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6fe23df6-0b08-431d-980f-aaa45d8d8093 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=75227) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 874.672234] env[75227]: CRITICAL neutron [None req-6fe23df6-0b08-431d-980f-aaa45d8d8093 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 874.672234] env[75227]: ERROR neutron Traceback (most recent call last): [ 874.672234] env[75227]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 874.672234] env[75227]: ERROR neutron sys.exit(main()) [ 874.672234] env[75227]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 874.672234] env[75227]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 874.672234] env[75227]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 874.672234] env[75227]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 874.672234] env[75227]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 874.672234] env[75227]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 874.672234] env[75227]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 874.672234] env[75227]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 874.672234] env[75227]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 874.672234] env[75227]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 874.672234] env[75227]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 874.672234] env[75227]: ERROR neutron response = self.get(path=path, params=params) [ 874.673005] env[75227]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 874.673005] env[75227]: ERROR neutron raise RuntimeError(msg, last_err) [ 874.673005] env[75227]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 874.673005] env[75227]: ERROR neutron [ 877.496324] env[75353]: INFO neutron.common.config [-] Logging enabled! [ 877.496718] env[75353]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 877.496933] env[75353]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=75353) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 877.497169] env[75353]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 877.497259] env[75353]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 877.498174] env[75353]: WARNING oslo_policy.policy [None req-a7182877-3fd9-42b9-b00d-708ca21cf005 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 877.498397] env[75353]: WARNING oslo_policy.policy [None req-a7182877-3fd9-42b9-b00d-708ca21cf005 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 877.499729] env[75353]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-a7182877-3fd9-42b9-b00d-708ca21cf005 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 877.515271] env[75353]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a7182877-3fd9-42b9-b00d-708ca21cf005 None None] Session token - acquiring [ 877.516195] env[75353]: DEBUG networking_nsxv3.common.locking [None req-a7182877-3fd9-42b9-b00d-708ca21cf005 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75353) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 877.516413] env[75353]: DEBUG oslo_concurrency.lockutils [None req-a7182877-3fd9-42b9-b00d-708ca21cf005 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75353) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 877.516533] env[75353]: DEBUG oslo_concurrency.lockutils [None req-a7182877-3fd9-42b9-b00d-708ca21cf005 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75353) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 877.516916] env[75353]: DEBUG oslo_concurrency.lockutils [None req-a7182877-3fd9-42b9-b00d-708ca21cf005 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75353) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 877.527068] env[75353]: DEBUG oslo_concurrency.lockutils [None req-a7182877-3fd9-42b9-b00d-708ca21cf005 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75353) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 877.527356] env[75353]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a7182877-3fd9-42b9-b00d-708ca21cf005 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 877.527505] env[75353]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a7182877-3fd9-42b9-b00d-708ca21cf005 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=75353) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 879.534045] env[75353]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a7182877-3fd9-42b9-b00d-708ca21cf005 None None] Session token - acquiring [ 879.534531] env[75353]: DEBUG networking_nsxv3.common.locking [None req-a7182877-3fd9-42b9-b00d-708ca21cf005 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75353) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 879.534725] env[75353]: DEBUG oslo_concurrency.lockutils [None req-a7182877-3fd9-42b9-b00d-708ca21cf005 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75353) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 879.534855] env[75353]: DEBUG oslo_concurrency.lockutils [None req-a7182877-3fd9-42b9-b00d-708ca21cf005 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75353) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 879.535131] env[75353]: DEBUG oslo_concurrency.lockutils [None req-a7182877-3fd9-42b9-b00d-708ca21cf005 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75353) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 879.544654] env[75353]: DEBUG oslo_concurrency.lockutils [None req-a7182877-3fd9-42b9-b00d-708ca21cf005 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75353) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 879.544927] env[75353]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a7182877-3fd9-42b9-b00d-708ca21cf005 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 879.545066] env[75353]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a7182877-3fd9-42b9-b00d-708ca21cf005 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=75353) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 881.547968] env[75353]: CRITICAL neutron [None req-a7182877-3fd9-42b9-b00d-708ca21cf005 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 881.547968] env[75353]: ERROR neutron Traceback (most recent call last): [ 881.547968] env[75353]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 881.547968] env[75353]: ERROR neutron sys.exit(main()) [ 881.547968] env[75353]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 881.547968] env[75353]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 881.547968] env[75353]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 881.547968] env[75353]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 881.547968] env[75353]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 881.547968] env[75353]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 881.547968] env[75353]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 881.547968] env[75353]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 881.547968] env[75353]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 881.547968] env[75353]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 881.547968] env[75353]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 881.547968] env[75353]: ERROR neutron response = self.get(path=path, params=params) [ 881.548777] env[75353]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 881.548777] env[75353]: ERROR neutron raise RuntimeError(msg, last_err) [ 881.548777] env[75353]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 881.548777] env[75353]: ERROR neutron [ 884.157748] env[75608]: INFO neutron.common.config [-] Logging enabled! [ 884.158101] env[75608]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 884.158256] env[75608]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=75608) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 884.158472] env[75608]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 884.158559] env[75608]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 884.159392] env[75608]: WARNING oslo_policy.policy [None req-c92559aa-e024-4a25-989d-4162d45b3073 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 884.159609] env[75608]: WARNING oslo_policy.policy [None req-c92559aa-e024-4a25-989d-4162d45b3073 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 884.160808] env[75608]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-c92559aa-e024-4a25-989d-4162d45b3073 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 884.175044] env[75608]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c92559aa-e024-4a25-989d-4162d45b3073 None None] Session token - acquiring [ 884.175845] env[75608]: DEBUG networking_nsxv3.common.locking [None req-c92559aa-e024-4a25-989d-4162d45b3073 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75608) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 884.176061] env[75608]: DEBUG oslo_concurrency.lockutils [None req-c92559aa-e024-4a25-989d-4162d45b3073 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75608) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 884.176190] env[75608]: DEBUG oslo_concurrency.lockutils [None req-c92559aa-e024-4a25-989d-4162d45b3073 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75608) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 884.176554] env[75608]: DEBUG oslo_concurrency.lockutils [None req-c92559aa-e024-4a25-989d-4162d45b3073 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75608) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 884.185492] env[75608]: DEBUG oslo_concurrency.lockutils [None req-c92559aa-e024-4a25-989d-4162d45b3073 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75608) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 884.185735] env[75608]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c92559aa-e024-4a25-989d-4162d45b3073 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 884.185877] env[75608]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c92559aa-e024-4a25-989d-4162d45b3073 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=75608) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 886.194091] env[75608]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c92559aa-e024-4a25-989d-4162d45b3073 None None] Session token - acquiring [ 886.194091] env[75608]: DEBUG networking_nsxv3.common.locking [None req-c92559aa-e024-4a25-989d-4162d45b3073 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75608) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 886.194091] env[75608]: DEBUG oslo_concurrency.lockutils [None req-c92559aa-e024-4a25-989d-4162d45b3073 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75608) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 886.194550] env[75608]: DEBUG oslo_concurrency.lockutils [None req-c92559aa-e024-4a25-989d-4162d45b3073 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75608) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 886.194550] env[75608]: DEBUG oslo_concurrency.lockutils [None req-c92559aa-e024-4a25-989d-4162d45b3073 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75608) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 886.202845] env[75608]: DEBUG oslo_concurrency.lockutils [None req-c92559aa-e024-4a25-989d-4162d45b3073 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75608) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 886.203204] env[75608]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c92559aa-e024-4a25-989d-4162d45b3073 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 886.203360] env[75608]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c92559aa-e024-4a25-989d-4162d45b3073 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=75608) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 888.205082] env[75608]: CRITICAL neutron [None req-c92559aa-e024-4a25-989d-4162d45b3073 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 888.205082] env[75608]: ERROR neutron Traceback (most recent call last): [ 888.205082] env[75608]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 888.205082] env[75608]: ERROR neutron sys.exit(main()) [ 888.205082] env[75608]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 888.205082] env[75608]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 888.205082] env[75608]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 888.205082] env[75608]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 888.205082] env[75608]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 888.205082] env[75608]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 888.205082] env[75608]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 888.205082] env[75608]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 888.205082] env[75608]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 888.205082] env[75608]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 888.205082] env[75608]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 888.205082] env[75608]: ERROR neutron response = self.get(path=path, params=params) [ 888.205829] env[75608]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 888.205829] env[75608]: ERROR neutron raise RuntimeError(msg, last_err) [ 888.205829] env[75608]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 888.205829] env[75608]: ERROR neutron [ 890.920198] env[75720]: INFO neutron.common.config [-] Logging enabled! [ 890.920555] env[75720]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 890.920725] env[75720]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=75720) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 890.920998] env[75720]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 890.921097] env[75720]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 890.922015] env[75720]: WARNING oslo_policy.policy [None req-593e929b-7631-470c-a292-f27f8762dcc5 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 890.922147] env[75720]: WARNING oslo_policy.policy [None req-593e929b-7631-470c-a292-f27f8762dcc5 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 890.923356] env[75720]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-593e929b-7631-470c-a292-f27f8762dcc5 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 890.937701] env[75720]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-593e929b-7631-470c-a292-f27f8762dcc5 None None] Session token - acquiring [ 890.938492] env[75720]: DEBUG networking_nsxv3.common.locking [None req-593e929b-7631-470c-a292-f27f8762dcc5 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75720) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 890.938698] env[75720]: DEBUG oslo_concurrency.lockutils [None req-593e929b-7631-470c-a292-f27f8762dcc5 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75720) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 890.938825] env[75720]: DEBUG oslo_concurrency.lockutils [None req-593e929b-7631-470c-a292-f27f8762dcc5 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75720) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 890.939201] env[75720]: DEBUG oslo_concurrency.lockutils [None req-593e929b-7631-470c-a292-f27f8762dcc5 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75720) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 890.948312] env[75720]: DEBUG oslo_concurrency.lockutils [None req-593e929b-7631-470c-a292-f27f8762dcc5 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75720) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 890.948568] env[75720]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-593e929b-7631-470c-a292-f27f8762dcc5 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 890.948715] env[75720]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-593e929b-7631-470c-a292-f27f8762dcc5 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=75720) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 892.965193] env[75720]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-593e929b-7631-470c-a292-f27f8762dcc5 None None] Session token - acquiring [ 892.965495] env[75720]: DEBUG networking_nsxv3.common.locking [None req-593e929b-7631-470c-a292-f27f8762dcc5 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75720) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 892.965669] env[75720]: DEBUG oslo_concurrency.lockutils [None req-593e929b-7631-470c-a292-f27f8762dcc5 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75720) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 892.965795] env[75720]: DEBUG oslo_concurrency.lockutils [None req-593e929b-7631-470c-a292-f27f8762dcc5 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75720) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 892.966131] env[75720]: DEBUG oslo_concurrency.lockutils [None req-593e929b-7631-470c-a292-f27f8762dcc5 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75720) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 892.978364] env[75720]: DEBUG oslo_concurrency.lockutils [None req-593e929b-7631-470c-a292-f27f8762dcc5 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75720) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 892.978570] env[75720]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-593e929b-7631-470c-a292-f27f8762dcc5 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 892.978712] env[75720]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-593e929b-7631-470c-a292-f27f8762dcc5 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=75720) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 894.981046] env[75720]: CRITICAL neutron [None req-593e929b-7631-470c-a292-f27f8762dcc5 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 894.981046] env[75720]: ERROR neutron Traceback (most recent call last): [ 894.981046] env[75720]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 894.981046] env[75720]: ERROR neutron sys.exit(main()) [ 894.981046] env[75720]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 894.981046] env[75720]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 894.981046] env[75720]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 894.981046] env[75720]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 894.981046] env[75720]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 894.981046] env[75720]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 894.981046] env[75720]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 894.981046] env[75720]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 894.981046] env[75720]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 894.981046] env[75720]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 894.981046] env[75720]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 894.981046] env[75720]: ERROR neutron response = self.get(path=path, params=params) [ 894.981793] env[75720]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 894.981793] env[75720]: ERROR neutron raise RuntimeError(msg, last_err) [ 894.981793] env[75720]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 894.981793] env[75720]: ERROR neutron [ 897.661474] env[75762]: INFO neutron.common.config [-] Logging enabled! [ 897.661884] env[75762]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 897.661972] env[75762]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=75762) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 897.662199] env[75762]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 897.662292] env[75762]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 897.663145] env[75762]: WARNING oslo_policy.policy [None req-10e0dfe5-7a50-45d5-8b62-5c2f7ab9ba62 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 897.663365] env[75762]: WARNING oslo_policy.policy [None req-10e0dfe5-7a50-45d5-8b62-5c2f7ab9ba62 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 897.664605] env[75762]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-10e0dfe5-7a50-45d5-8b62-5c2f7ab9ba62 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 897.679910] env[75762]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-10e0dfe5-7a50-45d5-8b62-5c2f7ab9ba62 None None] Session token - acquiring [ 897.680735] env[75762]: DEBUG networking_nsxv3.common.locking [None req-10e0dfe5-7a50-45d5-8b62-5c2f7ab9ba62 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75762) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 897.680952] env[75762]: DEBUG oslo_concurrency.lockutils [None req-10e0dfe5-7a50-45d5-8b62-5c2f7ab9ba62 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75762) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 897.681112] env[75762]: DEBUG oslo_concurrency.lockutils [None req-10e0dfe5-7a50-45d5-8b62-5c2f7ab9ba62 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75762) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 897.681494] env[75762]: DEBUG oslo_concurrency.lockutils [None req-10e0dfe5-7a50-45d5-8b62-5c2f7ab9ba62 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75762) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 897.690910] env[75762]: DEBUG oslo_concurrency.lockutils [None req-10e0dfe5-7a50-45d5-8b62-5c2f7ab9ba62 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75762) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 897.691162] env[75762]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-10e0dfe5-7a50-45d5-8b62-5c2f7ab9ba62 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 897.691305] env[75762]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-10e0dfe5-7a50-45d5-8b62-5c2f7ab9ba62 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=75762) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 899.697483] env[75762]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-10e0dfe5-7a50-45d5-8b62-5c2f7ab9ba62 None None] Session token - acquiring [ 899.697908] env[75762]: DEBUG networking_nsxv3.common.locking [None req-10e0dfe5-7a50-45d5-8b62-5c2f7ab9ba62 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75762) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 899.698212] env[75762]: DEBUG oslo_concurrency.lockutils [None req-10e0dfe5-7a50-45d5-8b62-5c2f7ab9ba62 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75762) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 899.698408] env[75762]: DEBUG oslo_concurrency.lockutils [None req-10e0dfe5-7a50-45d5-8b62-5c2f7ab9ba62 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75762) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 899.698710] env[75762]: DEBUG oslo_concurrency.lockutils [None req-10e0dfe5-7a50-45d5-8b62-5c2f7ab9ba62 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75762) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 899.707696] env[75762]: DEBUG oslo_concurrency.lockutils [None req-10e0dfe5-7a50-45d5-8b62-5c2f7ab9ba62 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75762) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 899.707938] env[75762]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-10e0dfe5-7a50-45d5-8b62-5c2f7ab9ba62 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 899.708243] env[75762]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-10e0dfe5-7a50-45d5-8b62-5c2f7ab9ba62 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=75762) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 901.711144] env[75762]: CRITICAL neutron [None req-10e0dfe5-7a50-45d5-8b62-5c2f7ab9ba62 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 901.711144] env[75762]: ERROR neutron Traceback (most recent call last): [ 901.711144] env[75762]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 901.711144] env[75762]: ERROR neutron sys.exit(main()) [ 901.711144] env[75762]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 901.711144] env[75762]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 901.711144] env[75762]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 901.711144] env[75762]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 901.711144] env[75762]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 901.711144] env[75762]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 901.711144] env[75762]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 901.711144] env[75762]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 901.711144] env[75762]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 901.711144] env[75762]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 901.711144] env[75762]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 901.711144] env[75762]: ERROR neutron response = self.get(path=path, params=params) [ 901.711928] env[75762]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 901.711928] env[75762]: ERROR neutron raise RuntimeError(msg, last_err) [ 901.711928] env[75762]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 901.711928] env[75762]: ERROR neutron [ 904.450978] env[75823]: INFO neutron.common.config [-] Logging enabled! [ 904.451283] env[75823]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 904.451507] env[75823]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=75823) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 904.451776] env[75823]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 904.451913] env[75823]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 904.452782] env[75823]: WARNING oslo_policy.policy [None req-54fbfe2a-5fa3-4bf4-a847-7559b7cafde8 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 904.453008] env[75823]: WARNING oslo_policy.policy [None req-54fbfe2a-5fa3-4bf4-a847-7559b7cafde8 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 904.454263] env[75823]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-54fbfe2a-5fa3-4bf4-a847-7559b7cafde8 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 904.476247] env[75823]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-54fbfe2a-5fa3-4bf4-a847-7559b7cafde8 None None] Session token - acquiring [ 904.477293] env[75823]: DEBUG networking_nsxv3.common.locking [None req-54fbfe2a-5fa3-4bf4-a847-7559b7cafde8 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75823) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 904.477683] env[75823]: DEBUG oslo_concurrency.lockutils [None req-54fbfe2a-5fa3-4bf4-a847-7559b7cafde8 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75823) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 904.477926] env[75823]: DEBUG oslo_concurrency.lockutils [None req-54fbfe2a-5fa3-4bf4-a847-7559b7cafde8 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75823) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 904.478394] env[75823]: DEBUG oslo_concurrency.lockutils [None req-54fbfe2a-5fa3-4bf4-a847-7559b7cafde8 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75823) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 904.490844] env[75823]: DEBUG oslo_concurrency.lockutils [None req-54fbfe2a-5fa3-4bf4-a847-7559b7cafde8 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75823) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 904.491288] env[75823]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-54fbfe2a-5fa3-4bf4-a847-7559b7cafde8 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 904.491455] env[75823]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-54fbfe2a-5fa3-4bf4-a847-7559b7cafde8 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=75823) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 906.497505] env[75823]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-54fbfe2a-5fa3-4bf4-a847-7559b7cafde8 None None] Session token - acquiring [ 906.497938] env[75823]: DEBUG networking_nsxv3.common.locking [None req-54fbfe2a-5fa3-4bf4-a847-7559b7cafde8 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75823) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 906.498024] env[75823]: DEBUG oslo_concurrency.lockutils [None req-54fbfe2a-5fa3-4bf4-a847-7559b7cafde8 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75823) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 906.498135] env[75823]: DEBUG oslo_concurrency.lockutils [None req-54fbfe2a-5fa3-4bf4-a847-7559b7cafde8 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75823) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 906.498471] env[75823]: DEBUG oslo_concurrency.lockutils [None req-54fbfe2a-5fa3-4bf4-a847-7559b7cafde8 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75823) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 906.509071] env[75823]: DEBUG oslo_concurrency.lockutils [None req-54fbfe2a-5fa3-4bf4-a847-7559b7cafde8 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75823) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 906.509460] env[75823]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-54fbfe2a-5fa3-4bf4-a847-7559b7cafde8 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 906.509662] env[75823]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-54fbfe2a-5fa3-4bf4-a847-7559b7cafde8 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=75823) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 908.512623] env[75823]: CRITICAL neutron [None req-54fbfe2a-5fa3-4bf4-a847-7559b7cafde8 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 908.512623] env[75823]: ERROR neutron Traceback (most recent call last): [ 908.512623] env[75823]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 908.512623] env[75823]: ERROR neutron sys.exit(main()) [ 908.512623] env[75823]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 908.512623] env[75823]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 908.512623] env[75823]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 908.512623] env[75823]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 908.512623] env[75823]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 908.512623] env[75823]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 908.512623] env[75823]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 908.512623] env[75823]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 908.512623] env[75823]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 908.512623] env[75823]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 908.512623] env[75823]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 908.512623] env[75823]: ERROR neutron response = self.get(path=path, params=params) [ 908.513380] env[75823]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 908.513380] env[75823]: ERROR neutron raise RuntimeError(msg, last_err) [ 908.513380] env[75823]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 908.513380] env[75823]: ERROR neutron [ 911.154748] env[75938]: INFO neutron.common.config [-] Logging enabled! [ 911.155165] env[75938]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 911.155279] env[75938]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=75938) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 911.155513] env[75938]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 911.155607] env[75938]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 911.156506] env[75938]: WARNING oslo_policy.policy [None req-f8669a43-780f-4002-8777-052bd2f40a1a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 911.156726] env[75938]: WARNING oslo_policy.policy [None req-f8669a43-780f-4002-8777-052bd2f40a1a None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 911.157965] env[75938]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-f8669a43-780f-4002-8777-052bd2f40a1a None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 911.175657] env[75938]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f8669a43-780f-4002-8777-052bd2f40a1a None None] Session token - acquiring [ 911.175657] env[75938]: DEBUG networking_nsxv3.common.locking [None req-f8669a43-780f-4002-8777-052bd2f40a1a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75938) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 911.175657] env[75938]: DEBUG oslo_concurrency.lockutils [None req-f8669a43-780f-4002-8777-052bd2f40a1a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75938) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 911.175802] env[75938]: DEBUG oslo_concurrency.lockutils [None req-f8669a43-780f-4002-8777-052bd2f40a1a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75938) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 911.175802] env[75938]: DEBUG oslo_concurrency.lockutils [None req-f8669a43-780f-4002-8777-052bd2f40a1a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75938) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 911.184454] env[75938]: DEBUG oslo_concurrency.lockutils [None req-f8669a43-780f-4002-8777-052bd2f40a1a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75938) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 911.184765] env[75938]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f8669a43-780f-4002-8777-052bd2f40a1a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 911.184931] env[75938]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f8669a43-780f-4002-8777-052bd2f40a1a None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=75938) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 913.190849] env[75938]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f8669a43-780f-4002-8777-052bd2f40a1a None None] Session token - acquiring [ 913.191198] env[75938]: DEBUG networking_nsxv3.common.locking [None req-f8669a43-780f-4002-8777-052bd2f40a1a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75938) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 913.191366] env[75938]: DEBUG oslo_concurrency.lockutils [None req-f8669a43-780f-4002-8777-052bd2f40a1a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75938) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 913.191456] env[75938]: DEBUG oslo_concurrency.lockutils [None req-f8669a43-780f-4002-8777-052bd2f40a1a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75938) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 913.191735] env[75938]: DEBUG oslo_concurrency.lockutils [None req-f8669a43-780f-4002-8777-052bd2f40a1a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75938) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 913.200893] env[75938]: DEBUG oslo_concurrency.lockutils [None req-f8669a43-780f-4002-8777-052bd2f40a1a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75938) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 913.201180] env[75938]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f8669a43-780f-4002-8777-052bd2f40a1a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 913.201327] env[75938]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f8669a43-780f-4002-8777-052bd2f40a1a None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=75938) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 915.204329] env[75938]: CRITICAL neutron [None req-f8669a43-780f-4002-8777-052bd2f40a1a None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 915.204329] env[75938]: ERROR neutron Traceback (most recent call last): [ 915.204329] env[75938]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 915.204329] env[75938]: ERROR neutron sys.exit(main()) [ 915.204329] env[75938]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 915.204329] env[75938]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 915.204329] env[75938]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 915.204329] env[75938]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 915.204329] env[75938]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 915.204329] env[75938]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 915.204329] env[75938]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 915.204329] env[75938]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 915.204329] env[75938]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 915.204329] env[75938]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 915.204329] env[75938]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 915.204329] env[75938]: ERROR neutron response = self.get(path=path, params=params) [ 915.205113] env[75938]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 915.205113] env[75938]: ERROR neutron raise RuntimeError(msg, last_err) [ 915.205113] env[75938]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 915.205113] env[75938]: ERROR neutron [ 918.036739] env[76213]: INFO neutron.common.config [-] Logging enabled! [ 918.037191] env[76213]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 918.037270] env[76213]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76213) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 918.037496] env[76213]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 918.037587] env[76213]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 918.038460] env[76213]: WARNING oslo_policy.policy [None req-a3aaa3c2-fe36-477a-933d-18e545c14d3e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 918.038667] env[76213]: WARNING oslo_policy.policy [None req-a3aaa3c2-fe36-477a-933d-18e545c14d3e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 918.039911] env[76213]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-a3aaa3c2-fe36-477a-933d-18e545c14d3e None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 918.054645] env[76213]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a3aaa3c2-fe36-477a-933d-18e545c14d3e None None] Session token - acquiring [ 918.055555] env[76213]: DEBUG networking_nsxv3.common.locking [None req-a3aaa3c2-fe36-477a-933d-18e545c14d3e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76213) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 918.055768] env[76213]: DEBUG oslo_concurrency.lockutils [None req-a3aaa3c2-fe36-477a-933d-18e545c14d3e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76213) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 918.055891] env[76213]: DEBUG oslo_concurrency.lockutils [None req-a3aaa3c2-fe36-477a-933d-18e545c14d3e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76213) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 918.056282] env[76213]: DEBUG oslo_concurrency.lockutils [None req-a3aaa3c2-fe36-477a-933d-18e545c14d3e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76213) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 918.065751] env[76213]: DEBUG oslo_concurrency.lockutils [None req-a3aaa3c2-fe36-477a-933d-18e545c14d3e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76213) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 918.066020] env[76213]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a3aaa3c2-fe36-477a-933d-18e545c14d3e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 918.066184] env[76213]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a3aaa3c2-fe36-477a-933d-18e545c14d3e None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76213) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 920.074117] env[76213]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a3aaa3c2-fe36-477a-933d-18e545c14d3e None None] Session token - acquiring [ 920.074475] env[76213]: DEBUG networking_nsxv3.common.locking [None req-a3aaa3c2-fe36-477a-933d-18e545c14d3e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76213) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 920.074680] env[76213]: DEBUG oslo_concurrency.lockutils [None req-a3aaa3c2-fe36-477a-933d-18e545c14d3e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76213) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 920.074812] env[76213]: DEBUG oslo_concurrency.lockutils [None req-a3aaa3c2-fe36-477a-933d-18e545c14d3e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76213) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 920.075085] env[76213]: DEBUG oslo_concurrency.lockutils [None req-a3aaa3c2-fe36-477a-933d-18e545c14d3e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76213) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 920.084280] env[76213]: DEBUG oslo_concurrency.lockutils [None req-a3aaa3c2-fe36-477a-933d-18e545c14d3e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76213) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 920.084508] env[76213]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a3aaa3c2-fe36-477a-933d-18e545c14d3e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 920.084648] env[76213]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a3aaa3c2-fe36-477a-933d-18e545c14d3e None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76213) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 922.085740] env[76213]: CRITICAL neutron [None req-a3aaa3c2-fe36-477a-933d-18e545c14d3e None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 922.085740] env[76213]: ERROR neutron Traceback (most recent call last): [ 922.085740] env[76213]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 922.085740] env[76213]: ERROR neutron sys.exit(main()) [ 922.085740] env[76213]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 922.085740] env[76213]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 922.085740] env[76213]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 922.085740] env[76213]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 922.085740] env[76213]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 922.085740] env[76213]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 922.085740] env[76213]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 922.085740] env[76213]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 922.085740] env[76213]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 922.085740] env[76213]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 922.085740] env[76213]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 922.085740] env[76213]: ERROR neutron response = self.get(path=path, params=params) [ 922.086536] env[76213]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 922.086536] env[76213]: ERROR neutron raise RuntimeError(msg, last_err) [ 922.086536] env[76213]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 922.086536] env[76213]: ERROR neutron [ 924.654605] env[76447]: INFO neutron.common.config [-] Logging enabled! [ 924.655034] env[76447]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 924.655121] env[76447]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76447) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 924.655357] env[76447]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 924.655449] env[76447]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 924.656973] env[76447]: WARNING oslo_policy.policy [None req-6830d8a0-68f7-4d70-a931-306688580ddb None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 924.656973] env[76447]: WARNING oslo_policy.policy [None req-6830d8a0-68f7-4d70-a931-306688580ddb None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 924.657698] env[76447]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-6830d8a0-68f7-4d70-a931-306688580ddb None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 924.672408] env[76447]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6830d8a0-68f7-4d70-a931-306688580ddb None None] Session token - acquiring [ 924.673182] env[76447]: DEBUG networking_nsxv3.common.locking [None req-6830d8a0-68f7-4d70-a931-306688580ddb None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76447) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 924.673399] env[76447]: DEBUG oslo_concurrency.lockutils [None req-6830d8a0-68f7-4d70-a931-306688580ddb None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76447) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 924.673525] env[76447]: DEBUG oslo_concurrency.lockutils [None req-6830d8a0-68f7-4d70-a931-306688580ddb None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76447) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 924.673928] env[76447]: DEBUG oslo_concurrency.lockutils [None req-6830d8a0-68f7-4d70-a931-306688580ddb None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76447) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 924.682967] env[76447]: DEBUG oslo_concurrency.lockutils [None req-6830d8a0-68f7-4d70-a931-306688580ddb None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76447) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 924.683217] env[76447]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6830d8a0-68f7-4d70-a931-306688580ddb None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 924.683366] env[76447]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6830d8a0-68f7-4d70-a931-306688580ddb None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76447) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 926.689536] env[76447]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6830d8a0-68f7-4d70-a931-306688580ddb None None] Session token - acquiring [ 926.689838] env[76447]: DEBUG networking_nsxv3.common.locking [None req-6830d8a0-68f7-4d70-a931-306688580ddb None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76447) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 926.690001] env[76447]: DEBUG oslo_concurrency.lockutils [None req-6830d8a0-68f7-4d70-a931-306688580ddb None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76447) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 926.690128] env[76447]: DEBUG oslo_concurrency.lockutils [None req-6830d8a0-68f7-4d70-a931-306688580ddb None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76447) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 926.690406] env[76447]: DEBUG oslo_concurrency.lockutils [None req-6830d8a0-68f7-4d70-a931-306688580ddb None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76447) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 926.699936] env[76447]: DEBUG oslo_concurrency.lockutils [None req-6830d8a0-68f7-4d70-a931-306688580ddb None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76447) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 926.700202] env[76447]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6830d8a0-68f7-4d70-a931-306688580ddb None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 926.700345] env[76447]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6830d8a0-68f7-4d70-a931-306688580ddb None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76447) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 928.703287] env[76447]: CRITICAL neutron [None req-6830d8a0-68f7-4d70-a931-306688580ddb None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 928.703287] env[76447]: ERROR neutron Traceback (most recent call last): [ 928.703287] env[76447]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 928.703287] env[76447]: ERROR neutron sys.exit(main()) [ 928.703287] env[76447]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 928.703287] env[76447]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 928.703287] env[76447]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 928.703287] env[76447]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 928.703287] env[76447]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 928.703287] env[76447]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 928.703287] env[76447]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 928.703287] env[76447]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 928.703287] env[76447]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 928.703287] env[76447]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 928.703287] env[76447]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 928.703287] env[76447]: ERROR neutron response = self.get(path=path, params=params) [ 928.704160] env[76447]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 928.704160] env[76447]: ERROR neutron raise RuntimeError(msg, last_err) [ 928.704160] env[76447]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 928.704160] env[76447]: ERROR neutron [ 931.375330] env[76571]: INFO neutron.common.config [-] Logging enabled! [ 931.375729] env[76571]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 931.375846] env[76571]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76571) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 931.376087] env[76571]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 931.376185] env[76571]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 931.377084] env[76571]: WARNING oslo_policy.policy [None req-7f702434-dfe5-4f5f-94a2-d713e3cb3cdd None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 931.377303] env[76571]: WARNING oslo_policy.policy [None req-7f702434-dfe5-4f5f-94a2-d713e3cb3cdd None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 931.378580] env[76571]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-7f702434-dfe5-4f5f-94a2-d713e3cb3cdd None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 931.393167] env[76571]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7f702434-dfe5-4f5f-94a2-d713e3cb3cdd None None] Session token - acquiring [ 931.394065] env[76571]: DEBUG networking_nsxv3.common.locking [None req-7f702434-dfe5-4f5f-94a2-d713e3cb3cdd None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76571) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 931.394273] env[76571]: DEBUG oslo_concurrency.lockutils [None req-7f702434-dfe5-4f5f-94a2-d713e3cb3cdd None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76571) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 931.394407] env[76571]: DEBUG oslo_concurrency.lockutils [None req-7f702434-dfe5-4f5f-94a2-d713e3cb3cdd None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76571) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 931.394824] env[76571]: DEBUG oslo_concurrency.lockutils [None req-7f702434-dfe5-4f5f-94a2-d713e3cb3cdd None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76571) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 931.403978] env[76571]: DEBUG oslo_concurrency.lockutils [None req-7f702434-dfe5-4f5f-94a2-d713e3cb3cdd None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76571) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 931.404258] env[76571]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7f702434-dfe5-4f5f-94a2-d713e3cb3cdd None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 931.404407] env[76571]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7f702434-dfe5-4f5f-94a2-d713e3cb3cdd None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76571) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 933.412907] env[76571]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7f702434-dfe5-4f5f-94a2-d713e3cb3cdd None None] Session token - acquiring [ 933.412907] env[76571]: DEBUG networking_nsxv3.common.locking [None req-7f702434-dfe5-4f5f-94a2-d713e3cb3cdd None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76571) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 933.412907] env[76571]: DEBUG oslo_concurrency.lockutils [None req-7f702434-dfe5-4f5f-94a2-d713e3cb3cdd None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76571) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 933.414730] env[76571]: DEBUG oslo_concurrency.lockutils [None req-7f702434-dfe5-4f5f-94a2-d713e3cb3cdd None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76571) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 933.414730] env[76571]: DEBUG oslo_concurrency.lockutils [None req-7f702434-dfe5-4f5f-94a2-d713e3cb3cdd None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76571) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 933.421530] env[76571]: DEBUG oslo_concurrency.lockutils [None req-7f702434-dfe5-4f5f-94a2-d713e3cb3cdd None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76571) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 933.421530] env[76571]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7f702434-dfe5-4f5f-94a2-d713e3cb3cdd None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 933.421530] env[76571]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-7f702434-dfe5-4f5f-94a2-d713e3cb3cdd None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76571) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 935.424421] env[76571]: CRITICAL neutron [None req-7f702434-dfe5-4f5f-94a2-d713e3cb3cdd None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 935.424421] env[76571]: ERROR neutron Traceback (most recent call last): [ 935.424421] env[76571]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 935.424421] env[76571]: ERROR neutron sys.exit(main()) [ 935.424421] env[76571]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 935.424421] env[76571]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 935.424421] env[76571]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 935.424421] env[76571]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 935.424421] env[76571]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 935.424421] env[76571]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 935.424421] env[76571]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 935.424421] env[76571]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 935.424421] env[76571]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 935.424421] env[76571]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 935.424421] env[76571]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 935.424421] env[76571]: ERROR neutron response = self.get(path=path, params=params) [ 935.425299] env[76571]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 935.425299] env[76571]: ERROR neutron raise RuntimeError(msg, last_err) [ 935.425299] env[76571]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 935.425299] env[76571]: ERROR neutron [ 938.030247] env[76621]: INFO neutron.common.config [-] Logging enabled! [ 938.030643] env[76621]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 938.030819] env[76621]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76621) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 938.031049] env[76621]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 938.031145] env[76621]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 938.032063] env[76621]: WARNING oslo_policy.policy [None req-43ab3760-b48d-400f-b84e-b30f19ef3e1e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 938.032285] env[76621]: WARNING oslo_policy.policy [None req-43ab3760-b48d-400f-b84e-b30f19ef3e1e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 938.033640] env[76621]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-43ab3760-b48d-400f-b84e-b30f19ef3e1e None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 938.048129] env[76621]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-43ab3760-b48d-400f-b84e-b30f19ef3e1e None None] Session token - acquiring [ 938.049810] env[76621]: DEBUG networking_nsxv3.common.locking [None req-43ab3760-b48d-400f-b84e-b30f19ef3e1e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76621) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 938.049810] env[76621]: DEBUG oslo_concurrency.lockutils [None req-43ab3760-b48d-400f-b84e-b30f19ef3e1e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76621) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 938.049975] env[76621]: DEBUG oslo_concurrency.lockutils [None req-43ab3760-b48d-400f-b84e-b30f19ef3e1e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76621) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 938.049975] env[76621]: DEBUG oslo_concurrency.lockutils [None req-43ab3760-b48d-400f-b84e-b30f19ef3e1e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76621) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 938.059768] env[76621]: DEBUG oslo_concurrency.lockutils [None req-43ab3760-b48d-400f-b84e-b30f19ef3e1e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76621) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 938.060037] env[76621]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-43ab3760-b48d-400f-b84e-b30f19ef3e1e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 938.060182] env[76621]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-43ab3760-b48d-400f-b84e-b30f19ef3e1e None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76621) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 940.068404] env[76621]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-43ab3760-b48d-400f-b84e-b30f19ef3e1e None None] Session token - acquiring [ 940.068724] env[76621]: DEBUG networking_nsxv3.common.locking [None req-43ab3760-b48d-400f-b84e-b30f19ef3e1e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76621) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 940.068795] env[76621]: DEBUG oslo_concurrency.lockutils [None req-43ab3760-b48d-400f-b84e-b30f19ef3e1e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76621) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 940.068880] env[76621]: DEBUG oslo_concurrency.lockutils [None req-43ab3760-b48d-400f-b84e-b30f19ef3e1e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76621) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 940.069156] env[76621]: DEBUG oslo_concurrency.lockutils [None req-43ab3760-b48d-400f-b84e-b30f19ef3e1e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76621) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 940.077892] env[76621]: DEBUG oslo_concurrency.lockutils [None req-43ab3760-b48d-400f-b84e-b30f19ef3e1e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76621) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 940.078100] env[76621]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-43ab3760-b48d-400f-b84e-b30f19ef3e1e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 940.078239] env[76621]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-43ab3760-b48d-400f-b84e-b30f19ef3e1e None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76621) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 942.080923] env[76621]: CRITICAL neutron [None req-43ab3760-b48d-400f-b84e-b30f19ef3e1e None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 942.080923] env[76621]: ERROR neutron Traceback (most recent call last): [ 942.080923] env[76621]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 942.080923] env[76621]: ERROR neutron sys.exit(main()) [ 942.080923] env[76621]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 942.080923] env[76621]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 942.080923] env[76621]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 942.080923] env[76621]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 942.080923] env[76621]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 942.080923] env[76621]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 942.080923] env[76621]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 942.080923] env[76621]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 942.080923] env[76621]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 942.080923] env[76621]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 942.080923] env[76621]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 942.080923] env[76621]: ERROR neutron response = self.get(path=path, params=params) [ 942.081945] env[76621]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 942.081945] env[76621]: ERROR neutron raise RuntimeError(msg, last_err) [ 942.081945] env[76621]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 942.081945] env[76621]: ERROR neutron [ 944.673137] env[76652]: INFO neutron.common.config [-] Logging enabled! [ 944.673535] env[76652]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 944.673731] env[76652]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76652) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 944.674702] env[76652]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 944.674702] env[76652]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 944.674877] env[76652]: WARNING oslo_policy.policy [None req-d2e345ec-00f9-4bc5-b70e-e96fc21a26d8 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 944.675217] env[76652]: WARNING oslo_policy.policy [None req-d2e345ec-00f9-4bc5-b70e-e96fc21a26d8 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 944.676758] env[76652]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-d2e345ec-00f9-4bc5-b70e-e96fc21a26d8 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 944.691327] env[76652]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d2e345ec-00f9-4bc5-b70e-e96fc21a26d8 None None] Session token - acquiring [ 944.692327] env[76652]: DEBUG networking_nsxv3.common.locking [None req-d2e345ec-00f9-4bc5-b70e-e96fc21a26d8 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76652) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 944.692529] env[76652]: DEBUG oslo_concurrency.lockutils [None req-d2e345ec-00f9-4bc5-b70e-e96fc21a26d8 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76652) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 944.692656] env[76652]: DEBUG oslo_concurrency.lockutils [None req-d2e345ec-00f9-4bc5-b70e-e96fc21a26d8 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76652) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 944.693023] env[76652]: DEBUG oslo_concurrency.lockutils [None req-d2e345ec-00f9-4bc5-b70e-e96fc21a26d8 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76652) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 944.702779] env[76652]: DEBUG oslo_concurrency.lockutils [None req-d2e345ec-00f9-4bc5-b70e-e96fc21a26d8 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76652) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 944.703042] env[76652]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d2e345ec-00f9-4bc5-b70e-e96fc21a26d8 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 944.703189] env[76652]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d2e345ec-00f9-4bc5-b70e-e96fc21a26d8 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76652) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 946.711207] env[76652]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d2e345ec-00f9-4bc5-b70e-e96fc21a26d8 None None] Session token - acquiring [ 946.711207] env[76652]: DEBUG networking_nsxv3.common.locking [None req-d2e345ec-00f9-4bc5-b70e-e96fc21a26d8 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76652) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 946.711207] env[76652]: DEBUG oslo_concurrency.lockutils [None req-d2e345ec-00f9-4bc5-b70e-e96fc21a26d8 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76652) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 946.712906] env[76652]: DEBUG oslo_concurrency.lockutils [None req-d2e345ec-00f9-4bc5-b70e-e96fc21a26d8 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76652) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 946.712906] env[76652]: DEBUG oslo_concurrency.lockutils [None req-d2e345ec-00f9-4bc5-b70e-e96fc21a26d8 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76652) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 946.720077] env[76652]: DEBUG oslo_concurrency.lockutils [None req-d2e345ec-00f9-4bc5-b70e-e96fc21a26d8 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76652) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 946.720077] env[76652]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d2e345ec-00f9-4bc5-b70e-e96fc21a26d8 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 946.720077] env[76652]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d2e345ec-00f9-4bc5-b70e-e96fc21a26d8 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76652) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 948.722954] env[76652]: CRITICAL neutron [None req-d2e345ec-00f9-4bc5-b70e-e96fc21a26d8 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 948.722954] env[76652]: ERROR neutron Traceback (most recent call last): [ 948.722954] env[76652]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 948.722954] env[76652]: ERROR neutron sys.exit(main()) [ 948.722954] env[76652]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 948.722954] env[76652]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 948.722954] env[76652]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 948.722954] env[76652]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 948.722954] env[76652]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 948.722954] env[76652]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 948.722954] env[76652]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 948.722954] env[76652]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 948.722954] env[76652]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 948.722954] env[76652]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 948.722954] env[76652]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 948.722954] env[76652]: ERROR neutron response = self.get(path=path, params=params) [ 948.724114] env[76652]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 948.724114] env[76652]: ERROR neutron raise RuntimeError(msg, last_err) [ 948.724114] env[76652]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 948.724114] env[76652]: ERROR neutron [ 951.394261] env[76848]: INFO neutron.common.config [-] Logging enabled! [ 951.394647] env[76848]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 951.394873] env[76848]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76848) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 951.395193] env[76848]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 951.395366] env[76848]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 951.396233] env[76848]: WARNING oslo_policy.policy [None req-1bfeacb6-c7f0-4275-a409-8036e547a492 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 951.396519] env[76848]: WARNING oslo_policy.policy [None req-1bfeacb6-c7f0-4275-a409-8036e547a492 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 951.397744] env[76848]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-1bfeacb6-c7f0-4275-a409-8036e547a492 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 951.411751] env[76848]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1bfeacb6-c7f0-4275-a409-8036e547a492 None None] Session token - acquiring [ 951.412544] env[76848]: DEBUG networking_nsxv3.common.locking [None req-1bfeacb6-c7f0-4275-a409-8036e547a492 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76848) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 951.412787] env[76848]: DEBUG oslo_concurrency.lockutils [None req-1bfeacb6-c7f0-4275-a409-8036e547a492 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76848) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 951.412951] env[76848]: DEBUG oslo_concurrency.lockutils [None req-1bfeacb6-c7f0-4275-a409-8036e547a492 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76848) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 951.413345] env[76848]: DEBUG oslo_concurrency.lockutils [None req-1bfeacb6-c7f0-4275-a409-8036e547a492 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76848) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 951.422094] env[76848]: DEBUG oslo_concurrency.lockutils [None req-1bfeacb6-c7f0-4275-a409-8036e547a492 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76848) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 951.422384] env[76848]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1bfeacb6-c7f0-4275-a409-8036e547a492 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 951.422569] env[76848]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1bfeacb6-c7f0-4275-a409-8036e547a492 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76848) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 953.428606] env[76848]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1bfeacb6-c7f0-4275-a409-8036e547a492 None None] Session token - acquiring [ 953.428966] env[76848]: DEBUG networking_nsxv3.common.locking [None req-1bfeacb6-c7f0-4275-a409-8036e547a492 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76848) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 953.429187] env[76848]: DEBUG oslo_concurrency.lockutils [None req-1bfeacb6-c7f0-4275-a409-8036e547a492 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76848) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 953.429317] env[76848]: DEBUG oslo_concurrency.lockutils [None req-1bfeacb6-c7f0-4275-a409-8036e547a492 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76848) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 953.429582] env[76848]: DEBUG oslo_concurrency.lockutils [None req-1bfeacb6-c7f0-4275-a409-8036e547a492 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76848) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 953.438624] env[76848]: DEBUG oslo_concurrency.lockutils [None req-1bfeacb6-c7f0-4275-a409-8036e547a492 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76848) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 953.438857] env[76848]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1bfeacb6-c7f0-4275-a409-8036e547a492 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 953.438997] env[76848]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1bfeacb6-c7f0-4275-a409-8036e547a492 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76848) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 955.440943] env[76848]: CRITICAL neutron [None req-1bfeacb6-c7f0-4275-a409-8036e547a492 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 955.440943] env[76848]: ERROR neutron Traceback (most recent call last): [ 955.440943] env[76848]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 955.440943] env[76848]: ERROR neutron sys.exit(main()) [ 955.440943] env[76848]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 955.440943] env[76848]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 955.440943] env[76848]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 955.440943] env[76848]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 955.440943] env[76848]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 955.440943] env[76848]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 955.440943] env[76848]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 955.440943] env[76848]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 955.440943] env[76848]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 955.440943] env[76848]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 955.440943] env[76848]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 955.440943] env[76848]: ERROR neutron response = self.get(path=path, params=params) [ 955.441893] env[76848]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 955.441893] env[76848]: ERROR neutron raise RuntimeError(msg, last_err) [ 955.441893] env[76848]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 955.441893] env[76848]: ERROR neutron [ 958.179146] env[76935]: INFO neutron.common.config [-] Logging enabled! [ 958.179437] env[76935]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 958.179782] env[76935]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76935) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 958.180093] env[76935]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 958.180210] env[76935]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 958.181382] env[76935]: WARNING oslo_policy.policy [None req-5ee558bf-2df8-4f1f-a24a-ec1fa9e4a22e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 958.181657] env[76935]: WARNING oslo_policy.policy [None req-5ee558bf-2df8-4f1f-a24a-ec1fa9e4a22e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 958.182864] env[76935]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-5ee558bf-2df8-4f1f-a24a-ec1fa9e4a22e None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 958.197090] env[76935]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5ee558bf-2df8-4f1f-a24a-ec1fa9e4a22e None None] Session token - acquiring [ 958.197923] env[76935]: DEBUG networking_nsxv3.common.locking [None req-5ee558bf-2df8-4f1f-a24a-ec1fa9e4a22e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76935) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 958.198158] env[76935]: DEBUG oslo_concurrency.lockutils [None req-5ee558bf-2df8-4f1f-a24a-ec1fa9e4a22e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76935) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 958.198158] env[76935]: DEBUG oslo_concurrency.lockutils [None req-5ee558bf-2df8-4f1f-a24a-ec1fa9e4a22e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76935) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 958.198614] env[76935]: DEBUG oslo_concurrency.lockutils [None req-5ee558bf-2df8-4f1f-a24a-ec1fa9e4a22e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76935) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 958.217453] env[76935]: DEBUG oslo_concurrency.lockutils [None req-5ee558bf-2df8-4f1f-a24a-ec1fa9e4a22e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76935) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 958.217453] env[76935]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5ee558bf-2df8-4f1f-a24a-ec1fa9e4a22e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 958.217453] env[76935]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5ee558bf-2df8-4f1f-a24a-ec1fa9e4a22e None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76935) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 960.216422] env[76935]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5ee558bf-2df8-4f1f-a24a-ec1fa9e4a22e None None] Session token - acquiring [ 960.216782] env[76935]: DEBUG networking_nsxv3.common.locking [None req-5ee558bf-2df8-4f1f-a24a-ec1fa9e4a22e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76935) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 960.217016] env[76935]: DEBUG oslo_concurrency.lockutils [None req-5ee558bf-2df8-4f1f-a24a-ec1fa9e4a22e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76935) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 960.217188] env[76935]: DEBUG oslo_concurrency.lockutils [None req-5ee558bf-2df8-4f1f-a24a-ec1fa9e4a22e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76935) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 960.217466] env[76935]: DEBUG oslo_concurrency.lockutils [None req-5ee558bf-2df8-4f1f-a24a-ec1fa9e4a22e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76935) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 960.226500] env[76935]: DEBUG oslo_concurrency.lockutils [None req-5ee558bf-2df8-4f1f-a24a-ec1fa9e4a22e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76935) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 960.226748] env[76935]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5ee558bf-2df8-4f1f-a24a-ec1fa9e4a22e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 960.226888] env[76935]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5ee558bf-2df8-4f1f-a24a-ec1fa9e4a22e None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76935) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 962.228894] env[76935]: CRITICAL neutron [None req-5ee558bf-2df8-4f1f-a24a-ec1fa9e4a22e None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 962.228894] env[76935]: ERROR neutron Traceback (most recent call last): [ 962.228894] env[76935]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 962.228894] env[76935]: ERROR neutron sys.exit(main()) [ 962.228894] env[76935]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 962.228894] env[76935]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 962.228894] env[76935]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 962.228894] env[76935]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 962.228894] env[76935]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 962.228894] env[76935]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 962.228894] env[76935]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 962.228894] env[76935]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 962.228894] env[76935]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 962.228894] env[76935]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 962.228894] env[76935]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 962.228894] env[76935]: ERROR neutron response = self.get(path=path, params=params) [ 962.229732] env[76935]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 962.229732] env[76935]: ERROR neutron raise RuntimeError(msg, last_err) [ 962.229732] env[76935]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 962.229732] env[76935]: ERROR neutron [ 964.819837] env[77045]: INFO neutron.common.config [-] Logging enabled! [ 964.820152] env[77045]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 964.820358] env[77045]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77045) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 964.820575] env[77045]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 964.820668] env[77045]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 964.821487] env[77045]: WARNING oslo_policy.policy [None req-e6522ab8-76a8-471e-b798-684b9b874583 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 964.821729] env[77045]: WARNING oslo_policy.policy [None req-e6522ab8-76a8-471e-b798-684b9b874583 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 964.822892] env[77045]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-e6522ab8-76a8-471e-b798-684b9b874583 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 964.836608] env[77045]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e6522ab8-76a8-471e-b798-684b9b874583 None None] Session token - acquiring [ 964.837352] env[77045]: DEBUG networking_nsxv3.common.locking [None req-e6522ab8-76a8-471e-b798-684b9b874583 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77045) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 964.837553] env[77045]: DEBUG oslo_concurrency.lockutils [None req-e6522ab8-76a8-471e-b798-684b9b874583 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77045) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 964.837680] env[77045]: DEBUG oslo_concurrency.lockutils [None req-e6522ab8-76a8-471e-b798-684b9b874583 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77045) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 964.838034] env[77045]: DEBUG oslo_concurrency.lockutils [None req-e6522ab8-76a8-471e-b798-684b9b874583 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77045) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 964.846882] env[77045]: DEBUG oslo_concurrency.lockutils [None req-e6522ab8-76a8-471e-b798-684b9b874583 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77045) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 964.847130] env[77045]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e6522ab8-76a8-471e-b798-684b9b874583 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 964.847279] env[77045]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e6522ab8-76a8-471e-b798-684b9b874583 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77045) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 966.855455] env[77045]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e6522ab8-76a8-471e-b798-684b9b874583 None None] Session token - acquiring [ 966.855987] env[77045]: DEBUG networking_nsxv3.common.locking [None req-e6522ab8-76a8-471e-b798-684b9b874583 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77045) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 966.855987] env[77045]: DEBUG oslo_concurrency.lockutils [None req-e6522ab8-76a8-471e-b798-684b9b874583 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77045) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 966.856140] env[77045]: DEBUG oslo_concurrency.lockutils [None req-e6522ab8-76a8-471e-b798-684b9b874583 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77045) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 966.856434] env[77045]: DEBUG oslo_concurrency.lockutils [None req-e6522ab8-76a8-471e-b798-684b9b874583 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77045) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 966.865423] env[77045]: DEBUG oslo_concurrency.lockutils [None req-e6522ab8-76a8-471e-b798-684b9b874583 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77045) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 966.865656] env[77045]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e6522ab8-76a8-471e-b798-684b9b874583 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 966.865793] env[77045]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e6522ab8-76a8-471e-b798-684b9b874583 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77045) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 968.868807] env[77045]: CRITICAL neutron [None req-e6522ab8-76a8-471e-b798-684b9b874583 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 968.868807] env[77045]: ERROR neutron Traceback (most recent call last): [ 968.868807] env[77045]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 968.868807] env[77045]: ERROR neutron sys.exit(main()) [ 968.868807] env[77045]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 968.868807] env[77045]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 968.868807] env[77045]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 968.868807] env[77045]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 968.868807] env[77045]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 968.868807] env[77045]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 968.868807] env[77045]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 968.868807] env[77045]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 968.868807] env[77045]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 968.868807] env[77045]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 968.868807] env[77045]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 968.868807] env[77045]: ERROR neutron response = self.get(path=path, params=params) [ 968.869528] env[77045]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 968.869528] env[77045]: ERROR neutron raise RuntimeError(msg, last_err) [ 968.869528] env[77045]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 968.869528] env[77045]: ERROR neutron [ 971.380692] env[77108]: INFO neutron.common.config [-] Logging enabled! [ 971.381029] env[77108]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 971.381156] env[77108]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77108) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 971.381375] env[77108]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 971.381460] env[77108]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 971.382276] env[77108]: WARNING oslo_policy.policy [None req-1cb6c1aa-c979-49af-8254-2f1bfd53b42e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 971.382488] env[77108]: WARNING oslo_policy.policy [None req-1cb6c1aa-c979-49af-8254-2f1bfd53b42e None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 971.383801] env[77108]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-1cb6c1aa-c979-49af-8254-2f1bfd53b42e None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 971.398335] env[77108]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1cb6c1aa-c979-49af-8254-2f1bfd53b42e None None] Session token - acquiring [ 971.399247] env[77108]: DEBUG networking_nsxv3.common.locking [None req-1cb6c1aa-c979-49af-8254-2f1bfd53b42e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77108) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 971.399333] env[77108]: DEBUG oslo_concurrency.lockutils [None req-1cb6c1aa-c979-49af-8254-2f1bfd53b42e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77108) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 971.399422] env[77108]: DEBUG oslo_concurrency.lockutils [None req-1cb6c1aa-c979-49af-8254-2f1bfd53b42e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77108) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 971.399857] env[77108]: DEBUG oslo_concurrency.lockutils [None req-1cb6c1aa-c979-49af-8254-2f1bfd53b42e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77108) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 971.419792] env[77108]: DEBUG oslo_concurrency.lockutils [None req-1cb6c1aa-c979-49af-8254-2f1bfd53b42e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77108) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 971.420967] env[77108]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1cb6c1aa-c979-49af-8254-2f1bfd53b42e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 971.420967] env[77108]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1cb6c1aa-c979-49af-8254-2f1bfd53b42e None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77108) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 973.426384] env[77108]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1cb6c1aa-c979-49af-8254-2f1bfd53b42e None None] Session token - acquiring [ 973.426886] env[77108]: DEBUG networking_nsxv3.common.locking [None req-1cb6c1aa-c979-49af-8254-2f1bfd53b42e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77108) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 973.427095] env[77108]: DEBUG oslo_concurrency.lockutils [None req-1cb6c1aa-c979-49af-8254-2f1bfd53b42e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77108) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 973.427224] env[77108]: DEBUG oslo_concurrency.lockutils [None req-1cb6c1aa-c979-49af-8254-2f1bfd53b42e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77108) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 973.427508] env[77108]: DEBUG oslo_concurrency.lockutils [None req-1cb6c1aa-c979-49af-8254-2f1bfd53b42e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77108) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 973.437010] env[77108]: DEBUG oslo_concurrency.lockutils [None req-1cb6c1aa-c979-49af-8254-2f1bfd53b42e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77108) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 973.437173] env[77108]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1cb6c1aa-c979-49af-8254-2f1bfd53b42e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 973.437312] env[77108]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1cb6c1aa-c979-49af-8254-2f1bfd53b42e None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77108) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 975.440253] env[77108]: CRITICAL neutron [None req-1cb6c1aa-c979-49af-8254-2f1bfd53b42e None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 975.440253] env[77108]: ERROR neutron Traceback (most recent call last): [ 975.440253] env[77108]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 975.440253] env[77108]: ERROR neutron sys.exit(main()) [ 975.440253] env[77108]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 975.440253] env[77108]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 975.440253] env[77108]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 975.440253] env[77108]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 975.440253] env[77108]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 975.440253] env[77108]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 975.440253] env[77108]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 975.440253] env[77108]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 975.440253] env[77108]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 975.440253] env[77108]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 975.440253] env[77108]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 975.440253] env[77108]: ERROR neutron response = self.get(path=path, params=params) [ 975.440923] env[77108]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 975.440923] env[77108]: ERROR neutron raise RuntimeError(msg, last_err) [ 975.440923] env[77108]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 975.440923] env[77108]: ERROR neutron [ 978.103408] env[77381]: INFO neutron.common.config [-] Logging enabled! [ 978.103769] env[77381]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 978.104014] env[77381]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77381) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 978.104296] env[77381]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 978.104392] env[77381]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 978.105234] env[77381]: WARNING oslo_policy.policy [None req-59796e93-9e58-4f1e-8a8f-ac97677617fd None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 978.105450] env[77381]: WARNING oslo_policy.policy [None req-59796e93-9e58-4f1e-8a8f-ac97677617fd None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 978.106684] env[77381]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-59796e93-9e58-4f1e-8a8f-ac97677617fd None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 978.120976] env[77381]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-59796e93-9e58-4f1e-8a8f-ac97677617fd None None] Session token - acquiring [ 978.121812] env[77381]: DEBUG networking_nsxv3.common.locking [None req-59796e93-9e58-4f1e-8a8f-ac97677617fd None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77381) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 978.122024] env[77381]: DEBUG oslo_concurrency.lockutils [None req-59796e93-9e58-4f1e-8a8f-ac97677617fd None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77381) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 978.122154] env[77381]: DEBUG oslo_concurrency.lockutils [None req-59796e93-9e58-4f1e-8a8f-ac97677617fd None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77381) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 978.122552] env[77381]: DEBUG oslo_concurrency.lockutils [None req-59796e93-9e58-4f1e-8a8f-ac97677617fd None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77381) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 978.131657] env[77381]: DEBUG oslo_concurrency.lockutils [None req-59796e93-9e58-4f1e-8a8f-ac97677617fd None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77381) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 978.131962] env[77381]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-59796e93-9e58-4f1e-8a8f-ac97677617fd None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 978.132151] env[77381]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-59796e93-9e58-4f1e-8a8f-ac97677617fd None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77381) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 980.140345] env[77381]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-59796e93-9e58-4f1e-8a8f-ac97677617fd None None] Session token - acquiring [ 980.140814] env[77381]: DEBUG networking_nsxv3.common.locking [None req-59796e93-9e58-4f1e-8a8f-ac97677617fd None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77381) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 980.140945] env[77381]: DEBUG oslo_concurrency.lockutils [None req-59796e93-9e58-4f1e-8a8f-ac97677617fd None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77381) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 980.141034] env[77381]: DEBUG oslo_concurrency.lockutils [None req-59796e93-9e58-4f1e-8a8f-ac97677617fd None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77381) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 980.141310] env[77381]: DEBUG oslo_concurrency.lockutils [None req-59796e93-9e58-4f1e-8a8f-ac97677617fd None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77381) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 980.150550] env[77381]: DEBUG oslo_concurrency.lockutils [None req-59796e93-9e58-4f1e-8a8f-ac97677617fd None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77381) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 980.150833] env[77381]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-59796e93-9e58-4f1e-8a8f-ac97677617fd None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 980.150980] env[77381]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-59796e93-9e58-4f1e-8a8f-ac97677617fd None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77381) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 982.153026] env[77381]: CRITICAL neutron [None req-59796e93-9e58-4f1e-8a8f-ac97677617fd None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 982.153026] env[77381]: ERROR neutron Traceback (most recent call last): [ 982.153026] env[77381]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 982.153026] env[77381]: ERROR neutron sys.exit(main()) [ 982.153026] env[77381]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 982.153026] env[77381]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 982.153026] env[77381]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 982.153026] env[77381]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 982.153026] env[77381]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 982.153026] env[77381]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 982.153026] env[77381]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 982.153026] env[77381]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 982.153026] env[77381]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 982.153026] env[77381]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 982.153026] env[77381]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 982.153026] env[77381]: ERROR neutron response = self.get(path=path, params=params) [ 982.154364] env[77381]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 982.154364] env[77381]: ERROR neutron raise RuntimeError(msg, last_err) [ 982.154364] env[77381]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 982.154364] env[77381]: ERROR neutron [ 984.513286] env[77424]: INFO neutron.common.config [-] Logging enabled! [ 984.513723] env[77424]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 984.513871] env[77424]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77424) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 984.514077] env[77424]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 984.514163] env[77424]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 984.515062] env[77424]: WARNING oslo_policy.policy [None req-68d2b320-c76e-4eca-afbb-551e522366e1 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 984.515308] env[77424]: WARNING oslo_policy.policy [None req-68d2b320-c76e-4eca-afbb-551e522366e1 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 984.516560] env[77424]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-68d2b320-c76e-4eca-afbb-551e522366e1 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 984.530939] env[77424]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-68d2b320-c76e-4eca-afbb-551e522366e1 None None] Session token - acquiring [ 984.531804] env[77424]: DEBUG networking_nsxv3.common.locking [None req-68d2b320-c76e-4eca-afbb-551e522366e1 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77424) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 984.532053] env[77424]: DEBUG oslo_concurrency.lockutils [None req-68d2b320-c76e-4eca-afbb-551e522366e1 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77424) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 984.532227] env[77424]: DEBUG oslo_concurrency.lockutils [None req-68d2b320-c76e-4eca-afbb-551e522366e1 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77424) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 984.532630] env[77424]: DEBUG oslo_concurrency.lockutils [None req-68d2b320-c76e-4eca-afbb-551e522366e1 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77424) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 984.541553] env[77424]: DEBUG oslo_concurrency.lockutils [None req-68d2b320-c76e-4eca-afbb-551e522366e1 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77424) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 984.541846] env[77424]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-68d2b320-c76e-4eca-afbb-551e522366e1 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 984.542001] env[77424]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-68d2b320-c76e-4eca-afbb-551e522366e1 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77424) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 986.548559] env[77424]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-68d2b320-c76e-4eca-afbb-551e522366e1 None None] Session token - acquiring [ 986.548993] env[77424]: DEBUG networking_nsxv3.common.locking [None req-68d2b320-c76e-4eca-afbb-551e522366e1 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77424) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 986.549111] env[77424]: DEBUG oslo_concurrency.lockutils [None req-68d2b320-c76e-4eca-afbb-551e522366e1 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77424) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 986.549199] env[77424]: DEBUG oslo_concurrency.lockutils [None req-68d2b320-c76e-4eca-afbb-551e522366e1 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77424) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 986.549475] env[77424]: DEBUG oslo_concurrency.lockutils [None req-68d2b320-c76e-4eca-afbb-551e522366e1 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77424) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 986.558760] env[77424]: DEBUG oslo_concurrency.lockutils [None req-68d2b320-c76e-4eca-afbb-551e522366e1 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77424) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 986.558986] env[77424]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-68d2b320-c76e-4eca-afbb-551e522366e1 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 986.559125] env[77424]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-68d2b320-c76e-4eca-afbb-551e522366e1 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77424) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 988.562154] env[77424]: CRITICAL neutron [None req-68d2b320-c76e-4eca-afbb-551e522366e1 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 988.562154] env[77424]: ERROR neutron Traceback (most recent call last): [ 988.562154] env[77424]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 988.562154] env[77424]: ERROR neutron sys.exit(main()) [ 988.562154] env[77424]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 988.562154] env[77424]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 988.562154] env[77424]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 988.562154] env[77424]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 988.562154] env[77424]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 988.562154] env[77424]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 988.562154] env[77424]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 988.562154] env[77424]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 988.562154] env[77424]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 988.562154] env[77424]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 988.562154] env[77424]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 988.562154] env[77424]: ERROR neutron response = self.get(path=path, params=params) [ 988.563395] env[77424]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 988.563395] env[77424]: ERROR neutron raise RuntimeError(msg, last_err) [ 988.563395] env[77424]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 988.563395] env[77424]: ERROR neutron [ 991.049132] env[77433]: INFO neutron.common.config [-] Logging enabled! [ 991.049574] env[77433]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 991.049676] env[77433]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77433) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 991.049882] env[77433]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 991.049978] env[77433]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 991.050836] env[77433]: WARNING oslo_policy.policy [None req-417ab522-44ff-4d62-9818-367796b40e64 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 991.051064] env[77433]: WARNING oslo_policy.policy [None req-417ab522-44ff-4d62-9818-367796b40e64 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 991.052324] env[77433]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-417ab522-44ff-4d62-9818-367796b40e64 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 991.068194] env[77433]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-417ab522-44ff-4d62-9818-367796b40e64 None None] Session token - acquiring [ 991.069008] env[77433]: DEBUG networking_nsxv3.common.locking [None req-417ab522-44ff-4d62-9818-367796b40e64 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77433) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 991.069217] env[77433]: DEBUG oslo_concurrency.lockutils [None req-417ab522-44ff-4d62-9818-367796b40e64 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77433) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 991.069350] env[77433]: DEBUG oslo_concurrency.lockutils [None req-417ab522-44ff-4d62-9818-367796b40e64 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77433) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 991.069722] env[77433]: DEBUG oslo_concurrency.lockutils [None req-417ab522-44ff-4d62-9818-367796b40e64 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77433) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 991.079608] env[77433]: DEBUG oslo_concurrency.lockutils [None req-417ab522-44ff-4d62-9818-367796b40e64 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77433) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 991.079860] env[77433]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-417ab522-44ff-4d62-9818-367796b40e64 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 991.080022] env[77433]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-417ab522-44ff-4d62-9818-367796b40e64 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77433) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 993.086238] env[77433]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-417ab522-44ff-4d62-9818-367796b40e64 None None] Session token - acquiring [ 993.086656] env[77433]: DEBUG networking_nsxv3.common.locking [None req-417ab522-44ff-4d62-9818-367796b40e64 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77433) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 993.086723] env[77433]: DEBUG oslo_concurrency.lockutils [None req-417ab522-44ff-4d62-9818-367796b40e64 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77433) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 993.086853] env[77433]: DEBUG oslo_concurrency.lockutils [None req-417ab522-44ff-4d62-9818-367796b40e64 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77433) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 993.087074] env[77433]: DEBUG oslo_concurrency.lockutils [None req-417ab522-44ff-4d62-9818-367796b40e64 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77433) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 993.096185] env[77433]: DEBUG oslo_concurrency.lockutils [None req-417ab522-44ff-4d62-9818-367796b40e64 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77433) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 993.096422] env[77433]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-417ab522-44ff-4d62-9818-367796b40e64 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 993.096563] env[77433]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-417ab522-44ff-4d62-9818-367796b40e64 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77433) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 995.099734] env[77433]: CRITICAL neutron [None req-417ab522-44ff-4d62-9818-367796b40e64 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 995.099734] env[77433]: ERROR neutron Traceback (most recent call last): [ 995.099734] env[77433]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 995.099734] env[77433]: ERROR neutron sys.exit(main()) [ 995.099734] env[77433]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 995.099734] env[77433]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 995.099734] env[77433]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 995.099734] env[77433]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 995.099734] env[77433]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 995.099734] env[77433]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 995.099734] env[77433]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 995.099734] env[77433]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 995.099734] env[77433]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 995.099734] env[77433]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 995.099734] env[77433]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 995.099734] env[77433]: ERROR neutron response = self.get(path=path, params=params) [ 995.100907] env[77433]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 995.100907] env[77433]: ERROR neutron raise RuntimeError(msg, last_err) [ 995.100907] env[77433]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 995.100907] env[77433]: ERROR neutron [ 997.643225] env[77441]: INFO neutron.common.config [-] Logging enabled! [ 997.643568] env[77441]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 997.643779] env[77441]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77441) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 997.644044] env[77441]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 997.644148] env[77441]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 997.645000] env[77441]: WARNING oslo_policy.policy [None req-a6fbfa96-14be-4f36-a92d-9b774858a600 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 997.645224] env[77441]: WARNING oslo_policy.policy [None req-a6fbfa96-14be-4f36-a92d-9b774858a600 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 997.646457] env[77441]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-a6fbfa96-14be-4f36-a92d-9b774858a600 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 997.662372] env[77441]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a6fbfa96-14be-4f36-a92d-9b774858a600 None None] Session token - acquiring [ 997.663252] env[77441]: DEBUG networking_nsxv3.common.locking [None req-a6fbfa96-14be-4f36-a92d-9b774858a600 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77441) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 997.663464] env[77441]: DEBUG oslo_concurrency.lockutils [None req-a6fbfa96-14be-4f36-a92d-9b774858a600 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77441) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 997.663592] env[77441]: DEBUG oslo_concurrency.lockutils [None req-a6fbfa96-14be-4f36-a92d-9b774858a600 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77441) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 997.664071] env[77441]: DEBUG oslo_concurrency.lockutils [None req-a6fbfa96-14be-4f36-a92d-9b774858a600 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77441) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 997.673214] env[77441]: DEBUG oslo_concurrency.lockutils [None req-a6fbfa96-14be-4f36-a92d-9b774858a600 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77441) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 997.673468] env[77441]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a6fbfa96-14be-4f36-a92d-9b774858a600 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 997.673616] env[77441]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a6fbfa96-14be-4f36-a92d-9b774858a600 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77441) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 999.679670] env[77441]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a6fbfa96-14be-4f36-a92d-9b774858a600 None None] Session token - acquiring [ 999.680067] env[77441]: DEBUG networking_nsxv3.common.locking [None req-a6fbfa96-14be-4f36-a92d-9b774858a600 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77441) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 999.680305] env[77441]: DEBUG oslo_concurrency.lockutils [None req-a6fbfa96-14be-4f36-a92d-9b774858a600 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77441) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 999.680436] env[77441]: DEBUG oslo_concurrency.lockutils [None req-a6fbfa96-14be-4f36-a92d-9b774858a600 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77441) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 999.680711] env[77441]: DEBUG oslo_concurrency.lockutils [None req-a6fbfa96-14be-4f36-a92d-9b774858a600 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77441) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 999.689787] env[77441]: DEBUG oslo_concurrency.lockutils [None req-a6fbfa96-14be-4f36-a92d-9b774858a600 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77441) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 999.690019] env[77441]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a6fbfa96-14be-4f36-a92d-9b774858a600 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 999.690160] env[77441]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a6fbfa96-14be-4f36-a92d-9b774858a600 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77441) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1001.692988] env[77441]: CRITICAL neutron [None req-a6fbfa96-14be-4f36-a92d-9b774858a600 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1001.692988] env[77441]: ERROR neutron Traceback (most recent call last): [ 1001.692988] env[77441]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 1001.692988] env[77441]: ERROR neutron sys.exit(main()) [ 1001.692988] env[77441]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 1001.692988] env[77441]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 1001.692988] env[77441]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 1001.692988] env[77441]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 1001.692988] env[77441]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 1001.692988] env[77441]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 1001.692988] env[77441]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 1001.692988] env[77441]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 1001.692988] env[77441]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 1001.692988] env[77441]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 1001.692988] env[77441]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 1001.692988] env[77441]: ERROR neutron response = self.get(path=path, params=params) [ 1001.694175] env[77441]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 1001.694175] env[77441]: ERROR neutron raise RuntimeError(msg, last_err) [ 1001.694175] env[77441]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1001.694175] env[77441]: ERROR neutron [ 1004.041869] env[77471]: INFO neutron.common.config [-] Logging enabled! [ 1004.042355] env[77471]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 1004.042432] env[77471]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77471) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 1004.042588] env[77471]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 1004.042679] env[77471]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 1004.043472] env[77471]: WARNING oslo_policy.policy [None req-6e72d77b-ae19-41f2-8e48-96d55e262c5b None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1004.043687] env[77471]: WARNING oslo_policy.policy [None req-6e72d77b-ae19-41f2-8e48-96d55e262c5b None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1004.044931] env[77471]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-6e72d77b-ae19-41f2-8e48-96d55e262c5b None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 1004.058849] env[77471]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6e72d77b-ae19-41f2-8e48-96d55e262c5b None None] Session token - acquiring [ 1004.059595] env[77471]: DEBUG networking_nsxv3.common.locking [None req-6e72d77b-ae19-41f2-8e48-96d55e262c5b None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77471) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1004.059794] env[77471]: DEBUG oslo_concurrency.lockutils [None req-6e72d77b-ae19-41f2-8e48-96d55e262c5b None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77471) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1004.059922] env[77471]: DEBUG oslo_concurrency.lockutils [None req-6e72d77b-ae19-41f2-8e48-96d55e262c5b None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77471) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1004.060299] env[77471]: DEBUG oslo_concurrency.lockutils [None req-6e72d77b-ae19-41f2-8e48-96d55e262c5b None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77471) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1004.069427] env[77471]: DEBUG oslo_concurrency.lockutils [None req-6e72d77b-ae19-41f2-8e48-96d55e262c5b None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77471) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1004.069671] env[77471]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6e72d77b-ae19-41f2-8e48-96d55e262c5b None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1004.069819] env[77471]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6e72d77b-ae19-41f2-8e48-96d55e262c5b None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77471) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1006.075842] env[77471]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6e72d77b-ae19-41f2-8e48-96d55e262c5b None None] Session token - acquiring [ 1006.076483] env[77471]: DEBUG networking_nsxv3.common.locking [None req-6e72d77b-ae19-41f2-8e48-96d55e262c5b None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77471) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1006.076483] env[77471]: DEBUG oslo_concurrency.lockutils [None req-6e72d77b-ae19-41f2-8e48-96d55e262c5b None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77471) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1006.076739] env[77471]: DEBUG oslo_concurrency.lockutils [None req-6e72d77b-ae19-41f2-8e48-96d55e262c5b None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77471) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1006.076796] env[77471]: DEBUG oslo_concurrency.lockutils [None req-6e72d77b-ae19-41f2-8e48-96d55e262c5b None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77471) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1006.085670] env[77471]: DEBUG oslo_concurrency.lockutils [None req-6e72d77b-ae19-41f2-8e48-96d55e262c5b None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77471) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1006.085901] env[77471]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6e72d77b-ae19-41f2-8e48-96d55e262c5b None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1006.086042] env[77471]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6e72d77b-ae19-41f2-8e48-96d55e262c5b None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77471) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1008.089439] env[77471]: CRITICAL neutron [None req-6e72d77b-ae19-41f2-8e48-96d55e262c5b None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1008.089439] env[77471]: ERROR neutron Traceback (most recent call last): [ 1008.089439] env[77471]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 1008.089439] env[77471]: ERROR neutron sys.exit(main()) [ 1008.089439] env[77471]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 1008.089439] env[77471]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 1008.089439] env[77471]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 1008.089439] env[77471]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 1008.089439] env[77471]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 1008.089439] env[77471]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 1008.089439] env[77471]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 1008.089439] env[77471]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 1008.089439] env[77471]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 1008.089439] env[77471]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 1008.089439] env[77471]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 1008.089439] env[77471]: ERROR neutron response = self.get(path=path, params=params) [ 1008.090250] env[77471]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 1008.090250] env[77471]: ERROR neutron raise RuntimeError(msg, last_err) [ 1008.090250] env[77471]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1008.090250] env[77471]: ERROR neutron [ 1010.621370] env[77478]: INFO neutron.common.config [-] Logging enabled! [ 1010.621788] env[77478]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 1010.621877] env[77478]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77478) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 1010.622119] env[77478]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 1010.622213] env[77478]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 1010.623023] env[77478]: WARNING oslo_policy.policy [None req-de3276de-fac8-4531-b62c-d64b903ea990 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1010.623272] env[77478]: WARNING oslo_policy.policy [None req-de3276de-fac8-4531-b62c-d64b903ea990 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1010.624471] env[77478]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-de3276de-fac8-4531-b62c-d64b903ea990 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 1010.638500] env[77478]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-de3276de-fac8-4531-b62c-d64b903ea990 None None] Session token - acquiring [ 1010.639233] env[77478]: DEBUG networking_nsxv3.common.locking [None req-de3276de-fac8-4531-b62c-d64b903ea990 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77478) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1010.639446] env[77478]: DEBUG oslo_concurrency.lockutils [None req-de3276de-fac8-4531-b62c-d64b903ea990 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77478) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1010.639597] env[77478]: DEBUG oslo_concurrency.lockutils [None req-de3276de-fac8-4531-b62c-d64b903ea990 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77478) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1010.639968] env[77478]: DEBUG oslo_concurrency.lockutils [None req-de3276de-fac8-4531-b62c-d64b903ea990 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77478) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1010.648862] env[77478]: DEBUG oslo_concurrency.lockutils [None req-de3276de-fac8-4531-b62c-d64b903ea990 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77478) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1010.649100] env[77478]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-de3276de-fac8-4531-b62c-d64b903ea990 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1010.649251] env[77478]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-de3276de-fac8-4531-b62c-d64b903ea990 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77478) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1012.656031] env[77478]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-de3276de-fac8-4531-b62c-d64b903ea990 None None] Session token - acquiring [ 1012.656031] env[77478]: DEBUG networking_nsxv3.common.locking [None req-de3276de-fac8-4531-b62c-d64b903ea990 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77478) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1012.656551] env[77478]: DEBUG oslo_concurrency.lockutils [None req-de3276de-fac8-4531-b62c-d64b903ea990 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77478) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1012.656551] env[77478]: DEBUG oslo_concurrency.lockutils [None req-de3276de-fac8-4531-b62c-d64b903ea990 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77478) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1012.656551] env[77478]: DEBUG oslo_concurrency.lockutils [None req-de3276de-fac8-4531-b62c-d64b903ea990 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77478) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1012.665499] env[77478]: DEBUG oslo_concurrency.lockutils [None req-de3276de-fac8-4531-b62c-d64b903ea990 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77478) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1012.665737] env[77478]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-de3276de-fac8-4531-b62c-d64b903ea990 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1012.665874] env[77478]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-de3276de-fac8-4531-b62c-d64b903ea990 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77478) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1014.668800] env[77478]: CRITICAL neutron [None req-de3276de-fac8-4531-b62c-d64b903ea990 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1014.668800] env[77478]: ERROR neutron Traceback (most recent call last): [ 1014.668800] env[77478]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 1014.668800] env[77478]: ERROR neutron sys.exit(main()) [ 1014.668800] env[77478]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 1014.668800] env[77478]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 1014.668800] env[77478]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 1014.668800] env[77478]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 1014.668800] env[77478]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 1014.668800] env[77478]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 1014.668800] env[77478]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 1014.668800] env[77478]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 1014.668800] env[77478]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 1014.668800] env[77478]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 1014.668800] env[77478]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 1014.668800] env[77478]: ERROR neutron response = self.get(path=path, params=params) [ 1014.669568] env[77478]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 1014.669568] env[77478]: ERROR neutron raise RuntimeError(msg, last_err) [ 1014.669568] env[77478]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1014.669568] env[77478]: ERROR neutron [ 1017.030129] env[77711]: INFO neutron.common.config [-] Logging enabled! [ 1017.030561] env[77711]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 1017.030608] env[77711]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77711) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 1017.030820] env[77711]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 1017.030911] env[77711]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 1017.031729] env[77711]: WARNING oslo_policy.policy [None req-6fa646d3-79e3-41c4-9c86-ddb20e6792f0 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1017.031948] env[77711]: WARNING oslo_policy.policy [None req-6fa646d3-79e3-41c4-9c86-ddb20e6792f0 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1017.033107] env[77711]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-6fa646d3-79e3-41c4-9c86-ddb20e6792f0 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 1017.046851] env[77711]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6fa646d3-79e3-41c4-9c86-ddb20e6792f0 None None] Session token - acquiring [ 1017.047586] env[77711]: DEBUG networking_nsxv3.common.locking [None req-6fa646d3-79e3-41c4-9c86-ddb20e6792f0 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77711) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1017.047781] env[77711]: DEBUG oslo_concurrency.lockutils [None req-6fa646d3-79e3-41c4-9c86-ddb20e6792f0 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77711) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1017.047909] env[77711]: DEBUG oslo_concurrency.lockutils [None req-6fa646d3-79e3-41c4-9c86-ddb20e6792f0 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77711) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1017.048298] env[77711]: DEBUG oslo_concurrency.lockutils [None req-6fa646d3-79e3-41c4-9c86-ddb20e6792f0 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77711) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1017.057130] env[77711]: DEBUG oslo_concurrency.lockutils [None req-6fa646d3-79e3-41c4-9c86-ddb20e6792f0 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77711) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1017.057375] env[77711]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6fa646d3-79e3-41c4-9c86-ddb20e6792f0 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1017.057519] env[77711]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6fa646d3-79e3-41c4-9c86-ddb20e6792f0 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77711) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1019.065475] env[77711]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6fa646d3-79e3-41c4-9c86-ddb20e6792f0 None None] Session token - acquiring [ 1019.065870] env[77711]: DEBUG networking_nsxv3.common.locking [None req-6fa646d3-79e3-41c4-9c86-ddb20e6792f0 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77711) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1019.066086] env[77711]: DEBUG oslo_concurrency.lockutils [None req-6fa646d3-79e3-41c4-9c86-ddb20e6792f0 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77711) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1019.066212] env[77711]: DEBUG oslo_concurrency.lockutils [None req-6fa646d3-79e3-41c4-9c86-ddb20e6792f0 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77711) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1019.066491] env[77711]: DEBUG oslo_concurrency.lockutils [None req-6fa646d3-79e3-41c4-9c86-ddb20e6792f0 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77711) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1019.075520] env[77711]: DEBUG oslo_concurrency.lockutils [None req-6fa646d3-79e3-41c4-9c86-ddb20e6792f0 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77711) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1019.075756] env[77711]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6fa646d3-79e3-41c4-9c86-ddb20e6792f0 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1019.075898] env[77711]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6fa646d3-79e3-41c4-9c86-ddb20e6792f0 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77711) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1021.078893] env[77711]: CRITICAL neutron [None req-6fa646d3-79e3-41c4-9c86-ddb20e6792f0 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1021.078893] env[77711]: ERROR neutron Traceback (most recent call last): [ 1021.078893] env[77711]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 1021.078893] env[77711]: ERROR neutron sys.exit(main()) [ 1021.078893] env[77711]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 1021.078893] env[77711]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 1021.078893] env[77711]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 1021.078893] env[77711]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 1021.078893] env[77711]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 1021.078893] env[77711]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 1021.078893] env[77711]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 1021.078893] env[77711]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 1021.078893] env[77711]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 1021.078893] env[77711]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 1021.078893] env[77711]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 1021.078893] env[77711]: ERROR neutron response = self.get(path=path, params=params) [ 1021.079766] env[77711]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 1021.079766] env[77711]: ERROR neutron raise RuntimeError(msg, last_err) [ 1021.079766] env[77711]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1021.079766] env[77711]: ERROR neutron [ 1023.560345] env[77756]: INFO neutron.common.config [-] Logging enabled! [ 1023.560750] env[77756]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 1023.560792] env[77756]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77756) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 1023.560992] env[77756]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 1023.561086] env[77756]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 1023.561909] env[77756]: WARNING oslo_policy.policy [None req-a4006e6e-425c-4d7f-87b1-4cd366c33155 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1023.562132] env[77756]: WARNING oslo_policy.policy [None req-a4006e6e-425c-4d7f-87b1-4cd366c33155 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1023.563270] env[77756]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-a4006e6e-425c-4d7f-87b1-4cd366c33155 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 1023.577173] env[77756]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a4006e6e-425c-4d7f-87b1-4cd366c33155 None None] Session token - acquiring [ 1023.577896] env[77756]: DEBUG networking_nsxv3.common.locking [None req-a4006e6e-425c-4d7f-87b1-4cd366c33155 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77756) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1023.578097] env[77756]: DEBUG oslo_concurrency.lockutils [None req-a4006e6e-425c-4d7f-87b1-4cd366c33155 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77756) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1023.578224] env[77756]: DEBUG oslo_concurrency.lockutils [None req-a4006e6e-425c-4d7f-87b1-4cd366c33155 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77756) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1023.578569] env[77756]: DEBUG oslo_concurrency.lockutils [None req-a4006e6e-425c-4d7f-87b1-4cd366c33155 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77756) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1023.587195] env[77756]: DEBUG oslo_concurrency.lockutils [None req-a4006e6e-425c-4d7f-87b1-4cd366c33155 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77756) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1023.587426] env[77756]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a4006e6e-425c-4d7f-87b1-4cd366c33155 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1023.587567] env[77756]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a4006e6e-425c-4d7f-87b1-4cd366c33155 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77756) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1025.593627] env[77756]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a4006e6e-425c-4d7f-87b1-4cd366c33155 None None] Session token - acquiring [ 1025.594032] env[77756]: DEBUG networking_nsxv3.common.locking [None req-a4006e6e-425c-4d7f-87b1-4cd366c33155 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77756) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1025.594181] env[77756]: DEBUG oslo_concurrency.lockutils [None req-a4006e6e-425c-4d7f-87b1-4cd366c33155 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77756) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1025.594286] env[77756]: DEBUG oslo_concurrency.lockutils [None req-a4006e6e-425c-4d7f-87b1-4cd366c33155 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77756) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1025.594552] env[77756]: DEBUG oslo_concurrency.lockutils [None req-a4006e6e-425c-4d7f-87b1-4cd366c33155 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77756) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1025.603488] env[77756]: DEBUG oslo_concurrency.lockutils [None req-a4006e6e-425c-4d7f-87b1-4cd366c33155 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77756) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1025.603736] env[77756]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a4006e6e-425c-4d7f-87b1-4cd366c33155 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1025.603911] env[77756]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a4006e6e-425c-4d7f-87b1-4cd366c33155 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77756) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1027.605011] env[77756]: CRITICAL neutron [None req-a4006e6e-425c-4d7f-87b1-4cd366c33155 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1027.605011] env[77756]: ERROR neutron Traceback (most recent call last): [ 1027.605011] env[77756]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 1027.605011] env[77756]: ERROR neutron sys.exit(main()) [ 1027.605011] env[77756]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 1027.605011] env[77756]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 1027.605011] env[77756]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 1027.605011] env[77756]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 1027.605011] env[77756]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 1027.605011] env[77756]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 1027.605011] env[77756]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 1027.605011] env[77756]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 1027.605011] env[77756]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 1027.605011] env[77756]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 1027.605011] env[77756]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 1027.605011] env[77756]: ERROR neutron response = self.get(path=path, params=params) [ 1027.605926] env[77756]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 1027.605926] env[77756]: ERROR neutron raise RuntimeError(msg, last_err) [ 1027.605926] env[77756]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1027.605926] env[77756]: ERROR neutron [ 1030.093596] env[77764]: INFO neutron.common.config [-] Logging enabled! [ 1030.094123] env[77764]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 1030.094123] env[77764]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77764) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 1030.094339] env[77764]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 1030.094434] env[77764]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 1030.095234] env[77764]: WARNING oslo_policy.policy [None req-a4de82d8-cf87-4bb8-8412-c003b538a9f9 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1030.095457] env[77764]: WARNING oslo_policy.policy [None req-a4de82d8-cf87-4bb8-8412-c003b538a9f9 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1030.096627] env[77764]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-a4de82d8-cf87-4bb8-8412-c003b538a9f9 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 1030.110873] env[77764]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a4de82d8-cf87-4bb8-8412-c003b538a9f9 None None] Session token - acquiring [ 1030.111606] env[77764]: DEBUG networking_nsxv3.common.locking [None req-a4de82d8-cf87-4bb8-8412-c003b538a9f9 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77764) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1030.111803] env[77764]: DEBUG oslo_concurrency.lockutils [None req-a4de82d8-cf87-4bb8-8412-c003b538a9f9 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77764) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1030.111932] env[77764]: DEBUG oslo_concurrency.lockutils [None req-a4de82d8-cf87-4bb8-8412-c003b538a9f9 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77764) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1030.112297] env[77764]: DEBUG oslo_concurrency.lockutils [None req-a4de82d8-cf87-4bb8-8412-c003b538a9f9 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77764) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1030.121084] env[77764]: DEBUG oslo_concurrency.lockutils [None req-a4de82d8-cf87-4bb8-8412-c003b538a9f9 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77764) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1030.121327] env[77764]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a4de82d8-cf87-4bb8-8412-c003b538a9f9 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1030.121473] env[77764]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a4de82d8-cf87-4bb8-8412-c003b538a9f9 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77764) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1032.127467] env[77764]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a4de82d8-cf87-4bb8-8412-c003b538a9f9 None None] Session token - acquiring [ 1032.128337] env[77764]: DEBUG networking_nsxv3.common.locking [None req-a4de82d8-cf87-4bb8-8412-c003b538a9f9 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77764) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1032.128758] env[77764]: DEBUG oslo_concurrency.lockutils [None req-a4de82d8-cf87-4bb8-8412-c003b538a9f9 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77764) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1032.129031] env[77764]: DEBUG oslo_concurrency.lockutils [None req-a4de82d8-cf87-4bb8-8412-c003b538a9f9 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77764) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1032.129422] env[77764]: DEBUG oslo_concurrency.lockutils [None req-a4de82d8-cf87-4bb8-8412-c003b538a9f9 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77764) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1032.138643] env[77764]: DEBUG oslo_concurrency.lockutils [None req-a4de82d8-cf87-4bb8-8412-c003b538a9f9 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77764) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1032.139020] env[77764]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a4de82d8-cf87-4bb8-8412-c003b538a9f9 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1032.139295] env[77764]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-a4de82d8-cf87-4bb8-8412-c003b538a9f9 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77764) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1034.144035] env[77764]: CRITICAL neutron [None req-a4de82d8-cf87-4bb8-8412-c003b538a9f9 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1034.144035] env[77764]: ERROR neutron Traceback (most recent call last): [ 1034.144035] env[77764]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 1034.144035] env[77764]: ERROR neutron sys.exit(main()) [ 1034.144035] env[77764]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 1034.144035] env[77764]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 1034.144035] env[77764]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 1034.144035] env[77764]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 1034.144035] env[77764]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 1034.144035] env[77764]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 1034.144035] env[77764]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 1034.144035] env[77764]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 1034.144035] env[77764]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 1034.144035] env[77764]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 1034.144035] env[77764]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 1034.144035] env[77764]: ERROR neutron response = self.get(path=path, params=params) [ 1034.145122] env[77764]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 1034.145122] env[77764]: ERROR neutron raise RuntimeError(msg, last_err) [ 1034.145122] env[77764]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1034.145122] env[77764]: ERROR neutron [ 1036.528380] env[77853]: INFO neutron.common.config [-] Logging enabled! [ 1036.528774] env[77853]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 1036.528889] env[77853]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77853) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 1036.529141] env[77853]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 1036.529239] env[77853]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 1036.530063] env[77853]: WARNING oslo_policy.policy [None req-892cae48-d0bf-4200-a149-c5d3dcf30ecb None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1036.530310] env[77853]: WARNING oslo_policy.policy [None req-892cae48-d0bf-4200-a149-c5d3dcf30ecb None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1036.531502] env[77853]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-892cae48-d0bf-4200-a149-c5d3dcf30ecb None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 1036.545618] env[77853]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-892cae48-d0bf-4200-a149-c5d3dcf30ecb None None] Session token - acquiring [ 1036.546383] env[77853]: DEBUG networking_nsxv3.common.locking [None req-892cae48-d0bf-4200-a149-c5d3dcf30ecb None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77853) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1036.546585] env[77853]: DEBUG oslo_concurrency.lockutils [None req-892cae48-d0bf-4200-a149-c5d3dcf30ecb None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77853) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1036.546713] env[77853]: DEBUG oslo_concurrency.lockutils [None req-892cae48-d0bf-4200-a149-c5d3dcf30ecb None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77853) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1036.547082] env[77853]: DEBUG oslo_concurrency.lockutils [None req-892cae48-d0bf-4200-a149-c5d3dcf30ecb None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77853) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1036.556052] env[77853]: DEBUG oslo_concurrency.lockutils [None req-892cae48-d0bf-4200-a149-c5d3dcf30ecb None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77853) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1036.556304] env[77853]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-892cae48-d0bf-4200-a149-c5d3dcf30ecb None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1036.556483] env[77853]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-892cae48-d0bf-4200-a149-c5d3dcf30ecb None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77853) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1038.562474] env[77853]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-892cae48-d0bf-4200-a149-c5d3dcf30ecb None None] Session token - acquiring [ 1038.562876] env[77853]: DEBUG networking_nsxv3.common.locking [None req-892cae48-d0bf-4200-a149-c5d3dcf30ecb None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77853) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1038.562970] env[77853]: DEBUG oslo_concurrency.lockutils [None req-892cae48-d0bf-4200-a149-c5d3dcf30ecb None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77853) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1038.563111] env[77853]: DEBUG oslo_concurrency.lockutils [None req-892cae48-d0bf-4200-a149-c5d3dcf30ecb None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77853) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1038.563389] env[77853]: DEBUG oslo_concurrency.lockutils [None req-892cae48-d0bf-4200-a149-c5d3dcf30ecb None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77853) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1038.572749] env[77853]: DEBUG oslo_concurrency.lockutils [None req-892cae48-d0bf-4200-a149-c5d3dcf30ecb None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77853) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1038.573031] env[77853]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-892cae48-d0bf-4200-a149-c5d3dcf30ecb None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1038.573190] env[77853]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-892cae48-d0bf-4200-a149-c5d3dcf30ecb None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77853) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1040.575678] env[77853]: CRITICAL neutron [None req-892cae48-d0bf-4200-a149-c5d3dcf30ecb None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1040.575678] env[77853]: ERROR neutron Traceback (most recent call last): [ 1040.575678] env[77853]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 1040.575678] env[77853]: ERROR neutron sys.exit(main()) [ 1040.575678] env[77853]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 1040.575678] env[77853]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 1040.575678] env[77853]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 1040.575678] env[77853]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 1040.575678] env[77853]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 1040.575678] env[77853]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 1040.575678] env[77853]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 1040.575678] env[77853]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 1040.575678] env[77853]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 1040.575678] env[77853]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 1040.575678] env[77853]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 1040.575678] env[77853]: ERROR neutron response = self.get(path=path, params=params) [ 1040.576460] env[77853]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 1040.576460] env[77853]: ERROR neutron raise RuntimeError(msg, last_err) [ 1040.576460] env[77853]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1040.576460] env[77853]: ERROR neutron [ 1042.985999] env[77862]: INFO neutron.common.config [-] Logging enabled! [ 1042.986455] env[77862]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 1042.986524] env[77862]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77862) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 1042.986720] env[77862]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 1042.986814] env[77862]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 1042.987640] env[77862]: WARNING oslo_policy.policy [None req-736aa2ea-9ab1-4ba8-850b-3100e150d39d None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1042.987858] env[77862]: WARNING oslo_policy.policy [None req-736aa2ea-9ab1-4ba8-850b-3100e150d39d None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1042.989117] env[77862]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-736aa2ea-9ab1-4ba8-850b-3100e150d39d None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 1043.003402] env[77862]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-736aa2ea-9ab1-4ba8-850b-3100e150d39d None None] Session token - acquiring [ 1043.004165] env[77862]: DEBUG networking_nsxv3.common.locking [None req-736aa2ea-9ab1-4ba8-850b-3100e150d39d None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77862) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1043.004372] env[77862]: DEBUG oslo_concurrency.lockutils [None req-736aa2ea-9ab1-4ba8-850b-3100e150d39d None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77862) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1043.004500] env[77862]: DEBUG oslo_concurrency.lockutils [None req-736aa2ea-9ab1-4ba8-850b-3100e150d39d None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77862) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1043.004865] env[77862]: DEBUG oslo_concurrency.lockutils [None req-736aa2ea-9ab1-4ba8-850b-3100e150d39d None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77862) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1043.013549] env[77862]: DEBUG oslo_concurrency.lockutils [None req-736aa2ea-9ab1-4ba8-850b-3100e150d39d None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77862) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1043.013800] env[77862]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-736aa2ea-9ab1-4ba8-850b-3100e150d39d None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1043.013950] env[77862]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-736aa2ea-9ab1-4ba8-850b-3100e150d39d None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77862) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1045.020162] env[77862]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-736aa2ea-9ab1-4ba8-850b-3100e150d39d None None] Session token - acquiring [ 1045.020779] env[77862]: DEBUG networking_nsxv3.common.locking [None req-736aa2ea-9ab1-4ba8-850b-3100e150d39d None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77862) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1045.021006] env[77862]: DEBUG oslo_concurrency.lockutils [None req-736aa2ea-9ab1-4ba8-850b-3100e150d39d None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77862) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1045.021136] env[77862]: DEBUG oslo_concurrency.lockutils [None req-736aa2ea-9ab1-4ba8-850b-3100e150d39d None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77862) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1045.021413] env[77862]: DEBUG oslo_concurrency.lockutils [None req-736aa2ea-9ab1-4ba8-850b-3100e150d39d None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77862) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1045.030914] env[77862]: DEBUG oslo_concurrency.lockutils [None req-736aa2ea-9ab1-4ba8-850b-3100e150d39d None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77862) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1045.031302] env[77862]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-736aa2ea-9ab1-4ba8-850b-3100e150d39d None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1045.031302] env[77862]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-736aa2ea-9ab1-4ba8-850b-3100e150d39d None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77862) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1047.048029] env[77862]: CRITICAL neutron [None req-736aa2ea-9ab1-4ba8-850b-3100e150d39d None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1047.048029] env[77862]: ERROR neutron Traceback (most recent call last): [ 1047.048029] env[77862]: ERROR neutron File "/opt/stack/data/venv/bin/neutron-nsxv3-agent", line 10, in [ 1047.048029] env[77862]: ERROR neutron sys.exit(main()) [ 1047.048029] env[77862]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 1047.048029] env[77862]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 1047.048029] env[77862]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 1047.048029] env[77862]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 1047.048029] env[77862]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 1047.048029] env[77862]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 1047.048029] env[77862]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 1047.048029] env[77862]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 1047.048029] env[77862]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 1047.048029] env[77862]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 1047.048029] env[77862]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 1047.048029] env[77862]: ERROR neutron response = self.get(path=path, params=params) [ 1047.048673] env[77862]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 1047.048673] env[77862]: ERROR neutron raise RuntimeError(msg, last_err) [ 1047.048673] env[77862]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 1047.048673] env[77862]: ERROR neutron [ 1049.522452] env[77954]: INFO neutron.common.config [-] Logging enabled! [ 1049.522781] env[77954]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-nsxv3-agent version 25.0.0.0b2.dev66 [ 1049.522961] env[77954]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77954) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 1049.523172] env[77954]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 1049.523268] env[77954]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 1049.524143] env[77954]: WARNING oslo_policy.policy [None req-4acc98a5-d301-4aa8-a547-8cffbdc7f2d3 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1049.524372] env[77954]: WARNING oslo_policy.policy [None req-4acc98a5-d301-4aa8-a547-8cffbdc7f2d3 None None] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html. [ 1049.525551] env[77954]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-4acc98a5-d301-4aa8-a547-8cffbdc7f2d3 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 1049.539886] env[77954]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4acc98a5-d301-4aa8-a547-8cffbdc7f2d3 None None] Session token - acquiring [ 1049.540641] env[77954]: DEBUG networking_nsxv3.common.locking [None req-4acc98a5-d301-4aa8-a547-8cffbdc7f2d3 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77954) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1049.540837] env[77954]: DEBUG oslo_concurrency.lockutils [None req-4acc98a5-d301-4aa8-a547-8cffbdc7f2d3 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1049.540964] env[77954]: DEBUG oslo_concurrency.lockutils [None req-4acc98a5-d301-4aa8-a547-8cffbdc7f2d3 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1049.541306] env[77954]: DEBUG oslo_concurrency.lockutils [None req-4acc98a5-d301-4aa8-a547-8cffbdc7f2d3 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1049.550021] env[77954]: DEBUG oslo_concurrency.lockutils [None req-4acc98a5-d301-4aa8-a547-8cffbdc7f2d3 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1049.550264] env[77954]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4acc98a5-d301-4aa8-a547-8cffbdc7f2d3 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1049.550414] env[77954]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4acc98a5-d301-4aa8-a547-8cffbdc7f2d3 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77954) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 1051.556441] env[77954]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4acc98a5-d301-4aa8-a547-8cffbdc7f2d3 None None] Session token - acquiring [ 1051.556779] env[77954]: DEBUG networking_nsxv3.common.locking [None req-4acc98a5-d301-4aa8-a547-8cffbdc7f2d3 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77954) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 1051.556971] env[77954]: DEBUG oslo_concurrency.lockutils [None req-4acc98a5-d301-4aa8-a547-8cffbdc7f2d3 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 1051.557043] env[77954]: DEBUG oslo_concurrency.lockutils [None req-4acc98a5-d301-4aa8-a547-8cffbdc7f2d3 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 1051.557323] env[77954]: DEBUG oslo_concurrency.lockutils [None req-4acc98a5-d301-4aa8-a547-8cffbdc7f2d3 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:321}} [ 1051.566863] env[77954]: DEBUG oslo_concurrency.lockutils [None req-4acc98a5-d301-4aa8-a547-8cffbdc7f2d3 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77954) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 1051.567118] env[77954]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4acc98a5-d301-4aa8-a547-8cffbdc7f2d3 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 1051.567259] env[77954]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4acc98a5-d301-4aa8-a547-8cffbdc7f2d3 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77954) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}}