[ 533.051039] env[60358]: INFO neutron.common.config [-] Logging enabled! [ 533.051534] env[60358]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 533.051534] env[60358]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=60358) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 533.051701] env[60358]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 533.051952] env[60358]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 533.052801] env[60358]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-de57091b-aab0-4e50-97f8-95a0228443ab None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 533.068514] env[60358]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-de57091b-aab0-4e50-97f8-95a0228443ab None None] Session token - acquiring [ 533.069230] env[60358]: DEBUG networking_nsxv3.common.locking [None req-de57091b-aab0-4e50-97f8-95a0228443ab None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=60358) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 533.069454] env[60358]: DEBUG oslo_concurrency.lockutils [None req-de57091b-aab0-4e50-97f8-95a0228443ab None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60358) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 533.069588] env[60358]: DEBUG oslo_concurrency.lockutils [None req-de57091b-aab0-4e50-97f8-95a0228443ab None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60358) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 533.069973] env[60358]: DEBUG oslo_concurrency.lockutils [None req-de57091b-aab0-4e50-97f8-95a0228443ab None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60358) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 533.079470] env[60358]: DEBUG oslo_concurrency.lockutils [None req-de57091b-aab0-4e50-97f8-95a0228443ab None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60358) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 533.079717] env[60358]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-de57091b-aab0-4e50-97f8-95a0228443ab None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 533.079864] env[60358]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-de57091b-aab0-4e50-97f8-95a0228443ab None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=60358) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 535.087728] env[60358]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-de57091b-aab0-4e50-97f8-95a0228443ab None None] Session token - acquiring [ 535.088127] env[60358]: DEBUG networking_nsxv3.common.locking [None req-de57091b-aab0-4e50-97f8-95a0228443ab None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=60358) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 535.088308] env[60358]: DEBUG oslo_concurrency.lockutils [None req-de57091b-aab0-4e50-97f8-95a0228443ab None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60358) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 535.088350] env[60358]: DEBUG oslo_concurrency.lockutils [None req-de57091b-aab0-4e50-97f8-95a0228443ab None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60358) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 535.088667] env[60358]: DEBUG oslo_concurrency.lockutils [None req-de57091b-aab0-4e50-97f8-95a0228443ab None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60358) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 535.097922] env[60358]: DEBUG oslo_concurrency.lockutils [None req-de57091b-aab0-4e50-97f8-95a0228443ab None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60358) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 535.098150] env[60358]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-de57091b-aab0-4e50-97f8-95a0228443ab None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 535.098291] env[60358]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-de57091b-aab0-4e50-97f8-95a0228443ab None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=60358) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 537.101058] env[60358]: CRITICAL neutron [None req-de57091b-aab0-4e50-97f8-95a0228443ab None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 537.101058] env[60358]: ERROR neutron Traceback (most recent call last): [ 537.101058] env[60358]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 537.101058] env[60358]: ERROR neutron sys.exit(main()) [ 537.101058] env[60358]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 537.101058] env[60358]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 537.101058] env[60358]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 537.101058] env[60358]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 537.101058] env[60358]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 537.101058] env[60358]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 537.101058] env[60358]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 537.101058] env[60358]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 537.101058] env[60358]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 537.101058] env[60358]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 537.101058] env[60358]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 537.101058] env[60358]: ERROR neutron response = self.get(path=path, params=params) [ 537.101750] env[60358]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 537.101750] env[60358]: ERROR neutron raise RuntimeError(msg, last_err) [ 537.101750] env[60358]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 537.101750] env[60358]: ERROR neutron [ 539.564956] env[60390]: INFO neutron.common.config [-] Logging enabled! [ 539.565345] env[60390]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 539.565380] env[60390]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=60390) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 539.565655] env[60390]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 539.565737] env[60390]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 539.566531] env[60390]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-c34378b5-7445-481c-b0e5-6ff18cd19aa2 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 539.579841] env[60390]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c34378b5-7445-481c-b0e5-6ff18cd19aa2 None None] Session token - acquiring [ 539.580643] env[60390]: DEBUG networking_nsxv3.common.locking [None req-c34378b5-7445-481c-b0e5-6ff18cd19aa2 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=60390) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 539.580839] env[60390]: DEBUG oslo_concurrency.lockutils [None req-c34378b5-7445-481c-b0e5-6ff18cd19aa2 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60390) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 539.580965] env[60390]: DEBUG oslo_concurrency.lockutils [None req-c34378b5-7445-481c-b0e5-6ff18cd19aa2 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60390) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 539.581305] env[60390]: DEBUG oslo_concurrency.lockutils [None req-c34378b5-7445-481c-b0e5-6ff18cd19aa2 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60390) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 539.590161] env[60390]: DEBUG oslo_concurrency.lockutils [None req-c34378b5-7445-481c-b0e5-6ff18cd19aa2 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60390) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 539.590419] env[60390]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c34378b5-7445-481c-b0e5-6ff18cd19aa2 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 539.590560] env[60390]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c34378b5-7445-481c-b0e5-6ff18cd19aa2 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=60390) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 541.597783] env[60390]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c34378b5-7445-481c-b0e5-6ff18cd19aa2 None None] Session token - acquiring [ 541.598100] env[60390]: DEBUG networking_nsxv3.common.locking [None req-c34378b5-7445-481c-b0e5-6ff18cd19aa2 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=60390) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 541.598260] env[60390]: DEBUG oslo_concurrency.lockutils [None req-c34378b5-7445-481c-b0e5-6ff18cd19aa2 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60390) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 541.598384] env[60390]: DEBUG oslo_concurrency.lockutils [None req-c34378b5-7445-481c-b0e5-6ff18cd19aa2 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60390) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 541.598670] env[60390]: DEBUG oslo_concurrency.lockutils [None req-c34378b5-7445-481c-b0e5-6ff18cd19aa2 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60390) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 541.608146] env[60390]: DEBUG oslo_concurrency.lockutils [None req-c34378b5-7445-481c-b0e5-6ff18cd19aa2 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60390) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 541.608407] env[60390]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c34378b5-7445-481c-b0e5-6ff18cd19aa2 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 541.608554] env[60390]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c34378b5-7445-481c-b0e5-6ff18cd19aa2 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=60390) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 543.611440] env[60390]: CRITICAL neutron [None req-c34378b5-7445-481c-b0e5-6ff18cd19aa2 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 543.611440] env[60390]: ERROR neutron Traceback (most recent call last): [ 543.611440] env[60390]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 543.611440] env[60390]: ERROR neutron sys.exit(main()) [ 543.611440] env[60390]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 543.611440] env[60390]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 543.611440] env[60390]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 543.611440] env[60390]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 543.611440] env[60390]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 543.611440] env[60390]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 543.611440] env[60390]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 543.611440] env[60390]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 543.611440] env[60390]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 543.611440] env[60390]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 543.611440] env[60390]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 543.611440] env[60390]: ERROR neutron response = self.get(path=path, params=params) [ 543.612179] env[60390]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 543.612179] env[60390]: ERROR neutron raise RuntimeError(msg, last_err) [ 543.612179] env[60390]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 543.612179] env[60390]: ERROR neutron [ 546.045462] env[60407]: INFO neutron.common.config [-] Logging enabled! [ 546.045822] env[60407]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 546.045939] env[60407]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=60407) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 546.046179] env[60407]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 546.046277] env[60407]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 546.047110] env[60407]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-6a8b68e7-0170-4898-bb66-501d0d68e74e None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 546.061097] env[60407]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6a8b68e7-0170-4898-bb66-501d0d68e74e None None] Session token - acquiring [ 546.061818] env[60407]: DEBUG networking_nsxv3.common.locking [None req-6a8b68e7-0170-4898-bb66-501d0d68e74e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=60407) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 546.062023] env[60407]: DEBUG oslo_concurrency.lockutils [None req-6a8b68e7-0170-4898-bb66-501d0d68e74e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60407) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 546.062150] env[60407]: DEBUG oslo_concurrency.lockutils [None req-6a8b68e7-0170-4898-bb66-501d0d68e74e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60407) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 546.062614] env[60407]: DEBUG oslo_concurrency.lockutils [None req-6a8b68e7-0170-4898-bb66-501d0d68e74e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60407) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 546.071397] env[60407]: DEBUG oslo_concurrency.lockutils [None req-6a8b68e7-0170-4898-bb66-501d0d68e74e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60407) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 546.071638] env[60407]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6a8b68e7-0170-4898-bb66-501d0d68e74e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 546.071807] env[60407]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6a8b68e7-0170-4898-bb66-501d0d68e74e None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=60407) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 548.078785] env[60407]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6a8b68e7-0170-4898-bb66-501d0d68e74e None None] Session token - acquiring [ 548.079200] env[60407]: DEBUG networking_nsxv3.common.locking [None req-6a8b68e7-0170-4898-bb66-501d0d68e74e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=60407) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 548.079200] env[60407]: DEBUG oslo_concurrency.lockutils [None req-6a8b68e7-0170-4898-bb66-501d0d68e74e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60407) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 548.079412] env[60407]: DEBUG oslo_concurrency.lockutils [None req-6a8b68e7-0170-4898-bb66-501d0d68e74e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60407) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 548.079597] env[60407]: DEBUG oslo_concurrency.lockutils [None req-6a8b68e7-0170-4898-bb66-501d0d68e74e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60407) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 548.088936] env[60407]: DEBUG oslo_concurrency.lockutils [None req-6a8b68e7-0170-4898-bb66-501d0d68e74e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60407) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 548.089210] env[60407]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6a8b68e7-0170-4898-bb66-501d0d68e74e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 548.089405] env[60407]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6a8b68e7-0170-4898-bb66-501d0d68e74e None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=60407) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 550.092380] env[60407]: CRITICAL neutron [None req-6a8b68e7-0170-4898-bb66-501d0d68e74e None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 550.092380] env[60407]: ERROR neutron Traceback (most recent call last): [ 550.092380] env[60407]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 550.092380] env[60407]: ERROR neutron sys.exit(main()) [ 550.092380] env[60407]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 550.092380] env[60407]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 550.092380] env[60407]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 550.092380] env[60407]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 550.092380] env[60407]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 550.092380] env[60407]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 550.092380] env[60407]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 550.092380] env[60407]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 550.092380] env[60407]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 550.092380] env[60407]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 550.092380] env[60407]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 550.092380] env[60407]: ERROR neutron response = self.get(path=path, params=params) [ 550.093126] env[60407]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 550.093126] env[60407]: ERROR neutron raise RuntimeError(msg, last_err) [ 550.093126] env[60407]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 550.093126] env[60407]: ERROR neutron [ 552.491384] env[60444]: INFO neutron.common.config [-] Logging enabled! [ 552.491816] env[60444]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 552.491875] env[60444]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=60444) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 552.492111] env[60444]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 552.492207] env[60444]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 552.493062] env[60444]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-29cdeef2-8bd5-4e63-8249-c22d6c351e62 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 552.506633] env[60444]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-29cdeef2-8bd5-4e63-8249-c22d6c351e62 None None] Session token - acquiring [ 552.507360] env[60444]: DEBUG networking_nsxv3.common.locking [None req-29cdeef2-8bd5-4e63-8249-c22d6c351e62 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=60444) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 552.507557] env[60444]: DEBUG oslo_concurrency.lockutils [None req-29cdeef2-8bd5-4e63-8249-c22d6c351e62 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60444) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 552.507685] env[60444]: DEBUG oslo_concurrency.lockutils [None req-29cdeef2-8bd5-4e63-8249-c22d6c351e62 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60444) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 552.508046] env[60444]: DEBUG oslo_concurrency.lockutils [None req-29cdeef2-8bd5-4e63-8249-c22d6c351e62 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60444) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 552.517329] env[60444]: DEBUG oslo_concurrency.lockutils [None req-29cdeef2-8bd5-4e63-8249-c22d6c351e62 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60444) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 552.517725] env[60444]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-29cdeef2-8bd5-4e63-8249-c22d6c351e62 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 552.517923] env[60444]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-29cdeef2-8bd5-4e63-8249-c22d6c351e62 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=60444) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 554.525755] env[60444]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-29cdeef2-8bd5-4e63-8249-c22d6c351e62 None None] Session token - acquiring [ 554.526130] env[60444]: DEBUG networking_nsxv3.common.locking [None req-29cdeef2-8bd5-4e63-8249-c22d6c351e62 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=60444) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 554.526559] env[60444]: DEBUG oslo_concurrency.lockutils [None req-29cdeef2-8bd5-4e63-8249-c22d6c351e62 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60444) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 554.526699] env[60444]: DEBUG oslo_concurrency.lockutils [None req-29cdeef2-8bd5-4e63-8249-c22d6c351e62 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60444) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 554.527009] env[60444]: DEBUG oslo_concurrency.lockutils [None req-29cdeef2-8bd5-4e63-8249-c22d6c351e62 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60444) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 554.536369] env[60444]: DEBUG oslo_concurrency.lockutils [None req-29cdeef2-8bd5-4e63-8249-c22d6c351e62 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60444) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 554.536624] env[60444]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-29cdeef2-8bd5-4e63-8249-c22d6c351e62 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 554.536769] env[60444]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-29cdeef2-8bd5-4e63-8249-c22d6c351e62 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=60444) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 556.539716] env[60444]: CRITICAL neutron [None req-29cdeef2-8bd5-4e63-8249-c22d6c351e62 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 556.539716] env[60444]: ERROR neutron Traceback (most recent call last): [ 556.539716] env[60444]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 556.539716] env[60444]: ERROR neutron sys.exit(main()) [ 556.539716] env[60444]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 556.539716] env[60444]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 556.539716] env[60444]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 556.539716] env[60444]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 556.539716] env[60444]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 556.539716] env[60444]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 556.539716] env[60444]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 556.539716] env[60444]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 556.539716] env[60444]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 556.539716] env[60444]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 556.539716] env[60444]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 556.539716] env[60444]: ERROR neutron response = self.get(path=path, params=params) [ 556.540482] env[60444]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 556.540482] env[60444]: ERROR neutron raise RuntimeError(msg, last_err) [ 556.540482] env[60444]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 556.540482] env[60444]: ERROR neutron [ 558.963628] env[60493]: INFO neutron.common.config [-] Logging enabled! [ 558.964024] env[60493]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 558.964066] env[60493]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=60493) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 558.964298] env[60493]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 558.964390] env[60493]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 558.965197] env[60493]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-165ba812-8f49-44b4-a260-6b9ca55bcb68 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 558.978840] env[60493]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-165ba812-8f49-44b4-a260-6b9ca55bcb68 None None] Session token - acquiring [ 558.979572] env[60493]: DEBUG networking_nsxv3.common.locking [None req-165ba812-8f49-44b4-a260-6b9ca55bcb68 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=60493) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 558.979767] env[60493]: DEBUG oslo_concurrency.lockutils [None req-165ba812-8f49-44b4-a260-6b9ca55bcb68 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60493) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 558.979894] env[60493]: DEBUG oslo_concurrency.lockutils [None req-165ba812-8f49-44b4-a260-6b9ca55bcb68 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60493) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 558.980250] env[60493]: DEBUG oslo_concurrency.lockutils [None req-165ba812-8f49-44b4-a260-6b9ca55bcb68 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60493) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 558.989238] env[60493]: DEBUG oslo_concurrency.lockutils [None req-165ba812-8f49-44b4-a260-6b9ca55bcb68 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60493) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 558.989530] env[60493]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-165ba812-8f49-44b4-a260-6b9ca55bcb68 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 558.989648] env[60493]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-165ba812-8f49-44b4-a260-6b9ca55bcb68 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=60493) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 560.997233] env[60493]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-165ba812-8f49-44b4-a260-6b9ca55bcb68 None None] Session token - acquiring [ 560.997589] env[60493]: DEBUG networking_nsxv3.common.locking [None req-165ba812-8f49-44b4-a260-6b9ca55bcb68 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=60493) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 560.997714] env[60493]: DEBUG oslo_concurrency.lockutils [None req-165ba812-8f49-44b4-a260-6b9ca55bcb68 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60493) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 560.997821] env[60493]: DEBUG oslo_concurrency.lockutils [None req-165ba812-8f49-44b4-a260-6b9ca55bcb68 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60493) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 560.998089] env[60493]: DEBUG oslo_concurrency.lockutils [None req-165ba812-8f49-44b4-a260-6b9ca55bcb68 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60493) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 561.007554] env[60493]: DEBUG oslo_concurrency.lockutils [None req-165ba812-8f49-44b4-a260-6b9ca55bcb68 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60493) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 561.007889] env[60493]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-165ba812-8f49-44b4-a260-6b9ca55bcb68 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 561.008094] env[60493]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-165ba812-8f49-44b4-a260-6b9ca55bcb68 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=60493) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 563.011016] env[60493]: CRITICAL neutron [None req-165ba812-8f49-44b4-a260-6b9ca55bcb68 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 563.011016] env[60493]: ERROR neutron Traceback (most recent call last): [ 563.011016] env[60493]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 563.011016] env[60493]: ERROR neutron sys.exit(main()) [ 563.011016] env[60493]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 563.011016] env[60493]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 563.011016] env[60493]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 563.011016] env[60493]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 563.011016] env[60493]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 563.011016] env[60493]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 563.011016] env[60493]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 563.011016] env[60493]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 563.011016] env[60493]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 563.011016] env[60493]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 563.011016] env[60493]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 563.011016] env[60493]: ERROR neutron response = self.get(path=path, params=params) [ 563.011752] env[60493]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 563.011752] env[60493]: ERROR neutron raise RuntimeError(msg, last_err) [ 563.011752] env[60493]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 563.011752] env[60493]: ERROR neutron [ 565.648794] env[60597]: INFO neutron.common.config [-] Logging enabled! [ 565.649110] env[60597]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 565.649312] env[60597]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=60597) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 565.649578] env[60597]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 565.649715] env[60597]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 565.650599] env[60597]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-e25c4944-aa8e-4e66-8859-a2f558df9fd4 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 565.664960] env[60597]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e25c4944-aa8e-4e66-8859-a2f558df9fd4 None None] Session token - acquiring [ 565.665863] env[60597]: DEBUG networking_nsxv3.common.locking [None req-e25c4944-aa8e-4e66-8859-a2f558df9fd4 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=60597) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 565.666079] env[60597]: DEBUG oslo_concurrency.lockutils [None req-e25c4944-aa8e-4e66-8859-a2f558df9fd4 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60597) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 565.666205] env[60597]: DEBUG oslo_concurrency.lockutils [None req-e25c4944-aa8e-4e66-8859-a2f558df9fd4 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60597) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 565.666604] env[60597]: DEBUG oslo_concurrency.lockutils [None req-e25c4944-aa8e-4e66-8859-a2f558df9fd4 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60597) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 565.681003] env[60597]: DEBUG oslo_concurrency.lockutils [None req-e25c4944-aa8e-4e66-8859-a2f558df9fd4 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60597) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 565.681297] env[60597]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e25c4944-aa8e-4e66-8859-a2f558df9fd4 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 565.681475] env[60597]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e25c4944-aa8e-4e66-8859-a2f558df9fd4 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=60597) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 567.689420] env[60597]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e25c4944-aa8e-4e66-8859-a2f558df9fd4 None None] Session token - acquiring [ 567.689730] env[60597]: DEBUG networking_nsxv3.common.locking [None req-e25c4944-aa8e-4e66-8859-a2f558df9fd4 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=60597) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 567.689914] env[60597]: DEBUG oslo_concurrency.lockutils [None req-e25c4944-aa8e-4e66-8859-a2f558df9fd4 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60597) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 567.690039] env[60597]: DEBUG oslo_concurrency.lockutils [None req-e25c4944-aa8e-4e66-8859-a2f558df9fd4 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60597) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 567.690326] env[60597]: DEBUG oslo_concurrency.lockutils [None req-e25c4944-aa8e-4e66-8859-a2f558df9fd4 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60597) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 567.702219] env[60597]: DEBUG oslo_concurrency.lockutils [None req-e25c4944-aa8e-4e66-8859-a2f558df9fd4 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60597) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 567.702383] env[60597]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e25c4944-aa8e-4e66-8859-a2f558df9fd4 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 567.702822] env[60597]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e25c4944-aa8e-4e66-8859-a2f558df9fd4 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=60597) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 569.704905] env[60597]: CRITICAL neutron [None req-e25c4944-aa8e-4e66-8859-a2f558df9fd4 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 569.704905] env[60597]: ERROR neutron Traceback (most recent call last): [ 569.704905] env[60597]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 569.704905] env[60597]: ERROR neutron sys.exit(main()) [ 569.704905] env[60597]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 569.704905] env[60597]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 569.704905] env[60597]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 569.704905] env[60597]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 569.704905] env[60597]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 569.704905] env[60597]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 569.704905] env[60597]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 569.704905] env[60597]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 569.704905] env[60597]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 569.704905] env[60597]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 569.704905] env[60597]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 569.704905] env[60597]: ERROR neutron response = self.get(path=path, params=params) [ 569.705533] env[60597]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 569.705533] env[60597]: ERROR neutron raise RuntimeError(msg, last_err) [ 569.705533] env[60597]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 569.705533] env[60597]: ERROR neutron [ 572.508938] env[60960]: INFO neutron.common.config [-] Logging enabled! [ 572.509174] env[60960]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 572.509456] env[60960]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=60960) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 572.509690] env[60960]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 572.509781] env[60960]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 572.510722] env[60960]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-1d7a11a4-c654-4779-a657-b288da5d5bec None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 572.526083] env[60960]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1d7a11a4-c654-4779-a657-b288da5d5bec None None] Session token - acquiring [ 572.527044] env[60960]: DEBUG networking_nsxv3.common.locking [None req-1d7a11a4-c654-4779-a657-b288da5d5bec None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=60960) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 572.527424] env[60960]: DEBUG oslo_concurrency.lockutils [None req-1d7a11a4-c654-4779-a657-b288da5d5bec None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60960) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 572.527709] env[60960]: DEBUG oslo_concurrency.lockutils [None req-1d7a11a4-c654-4779-a657-b288da5d5bec None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60960) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 572.528216] env[60960]: DEBUG oslo_concurrency.lockutils [None req-1d7a11a4-c654-4779-a657-b288da5d5bec None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60960) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 572.539553] env[60960]: DEBUG oslo_concurrency.lockutils [None req-1d7a11a4-c654-4779-a657-b288da5d5bec None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60960) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 572.539869] env[60960]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1d7a11a4-c654-4779-a657-b288da5d5bec None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 572.540093] env[60960]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1d7a11a4-c654-4779-a657-b288da5d5bec None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=60960) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 574.548098] env[60960]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1d7a11a4-c654-4779-a657-b288da5d5bec None None] Session token - acquiring [ 574.548376] env[60960]: DEBUG networking_nsxv3.common.locking [None req-1d7a11a4-c654-4779-a657-b288da5d5bec None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=60960) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 574.548517] env[60960]: DEBUG oslo_concurrency.lockutils [None req-1d7a11a4-c654-4779-a657-b288da5d5bec None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60960) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 574.548619] env[60960]: DEBUG oslo_concurrency.lockutils [None req-1d7a11a4-c654-4779-a657-b288da5d5bec None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60960) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 574.548969] env[60960]: DEBUG oslo_concurrency.lockutils [None req-1d7a11a4-c654-4779-a657-b288da5d5bec None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60960) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 574.558646] env[60960]: DEBUG oslo_concurrency.lockutils [None req-1d7a11a4-c654-4779-a657-b288da5d5bec None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=60960) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 574.558883] env[60960]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1d7a11a4-c654-4779-a657-b288da5d5bec None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 574.559023] env[60960]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1d7a11a4-c654-4779-a657-b288da5d5bec None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=60960) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 576.561730] env[60960]: CRITICAL neutron [None req-1d7a11a4-c654-4779-a657-b288da5d5bec None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 576.561730] env[60960]: ERROR neutron Traceback (most recent call last): [ 576.561730] env[60960]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 576.561730] env[60960]: ERROR neutron sys.exit(main()) [ 576.561730] env[60960]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 576.561730] env[60960]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 576.561730] env[60960]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 576.561730] env[60960]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 576.561730] env[60960]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 576.561730] env[60960]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 576.561730] env[60960]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 576.561730] env[60960]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 576.561730] env[60960]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 576.561730] env[60960]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 576.561730] env[60960]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 576.561730] env[60960]: ERROR neutron response = self.get(path=path, params=params) [ 576.562357] env[60960]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 576.562357] env[60960]: ERROR neutron raise RuntimeError(msg, last_err) [ 576.562357] env[60960]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 576.562357] env[60960]: ERROR neutron [ 579.304761] env[61521]: INFO neutron.common.config [-] Logging enabled! [ 579.304982] env[61521]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 579.305517] env[61521]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=61521) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 579.306050] env[61521]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 579.306050] env[61521]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 579.306976] env[61521]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-008d5c05-27e7-4629-a068-962b29081da1 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 579.327708] env[61521]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-008d5c05-27e7-4629-a068-962b29081da1 None None] Session token - acquiring [ 579.328570] env[61521]: DEBUG networking_nsxv3.common.locking [None req-008d5c05-27e7-4629-a068-962b29081da1 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=61521) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 579.328774] env[61521]: DEBUG oslo_concurrency.lockutils [None req-008d5c05-27e7-4629-a068-962b29081da1 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61521) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 579.328899] env[61521]: DEBUG oslo_concurrency.lockutils [None req-008d5c05-27e7-4629-a068-962b29081da1 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61521) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 579.329262] env[61521]: DEBUG oslo_concurrency.lockutils [None req-008d5c05-27e7-4629-a068-962b29081da1 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61521) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 579.345034] env[61521]: DEBUG oslo_concurrency.lockutils [None req-008d5c05-27e7-4629-a068-962b29081da1 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61521) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 579.345176] env[61521]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-008d5c05-27e7-4629-a068-962b29081da1 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 579.345301] env[61521]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-008d5c05-27e7-4629-a068-962b29081da1 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=61521) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 581.354766] env[61521]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-008d5c05-27e7-4629-a068-962b29081da1 None None] Session token - acquiring [ 581.355064] env[61521]: DEBUG networking_nsxv3.common.locking [None req-008d5c05-27e7-4629-a068-962b29081da1 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=61521) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 581.355254] env[61521]: DEBUG oslo_concurrency.lockutils [None req-008d5c05-27e7-4629-a068-962b29081da1 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61521) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 581.355385] env[61521]: DEBUG oslo_concurrency.lockutils [None req-008d5c05-27e7-4629-a068-962b29081da1 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61521) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 581.355696] env[61521]: DEBUG oslo_concurrency.lockutils [None req-008d5c05-27e7-4629-a068-962b29081da1 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61521) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 581.374019] env[61521]: DEBUG oslo_concurrency.lockutils [None req-008d5c05-27e7-4629-a068-962b29081da1 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=61521) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 581.376320] env[61521]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-008d5c05-27e7-4629-a068-962b29081da1 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 581.376714] env[61521]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-008d5c05-27e7-4629-a068-962b29081da1 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=61521) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 583.379770] env[61521]: CRITICAL neutron [None req-008d5c05-27e7-4629-a068-962b29081da1 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 583.379770] env[61521]: ERROR neutron Traceback (most recent call last): [ 583.379770] env[61521]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 583.379770] env[61521]: ERROR neutron sys.exit(main()) [ 583.379770] env[61521]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 583.379770] env[61521]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 583.379770] env[61521]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 583.379770] env[61521]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 583.379770] env[61521]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 583.379770] env[61521]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 583.379770] env[61521]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 583.379770] env[61521]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 583.379770] env[61521]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 583.379770] env[61521]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 583.379770] env[61521]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 583.379770] env[61521]: ERROR neutron response = self.get(path=path, params=params) [ 583.380507] env[61521]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 583.380507] env[61521]: ERROR neutron raise RuntimeError(msg, last_err) [ 583.380507] env[61521]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 583.380507] env[61521]: ERROR neutron [ 586.283073] env[62214]: INFO neutron.common.config [-] Logging enabled! [ 586.283319] env[62214]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 586.283598] env[62214]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=62214) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 586.283855] env[62214]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 586.283954] env[62214]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 586.285137] env[62214]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-5f3b4b5f-75eb-45f5-83b8-47ae2508461e None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 586.306225] env[62214]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5f3b4b5f-75eb-45f5-83b8-47ae2508461e None None] Session token - acquiring [ 586.307212] env[62214]: DEBUG networking_nsxv3.common.locking [None req-5f3b4b5f-75eb-45f5-83b8-47ae2508461e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62214) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 586.307343] env[62214]: DEBUG oslo_concurrency.lockutils [None req-5f3b4b5f-75eb-45f5-83b8-47ae2508461e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62214) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 586.307399] env[62214]: DEBUG oslo_concurrency.lockutils [None req-5f3b4b5f-75eb-45f5-83b8-47ae2508461e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62214) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 586.307750] env[62214]: DEBUG oslo_concurrency.lockutils [None req-5f3b4b5f-75eb-45f5-83b8-47ae2508461e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62214) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 586.317495] env[62214]: DEBUG oslo_concurrency.lockutils [None req-5f3b4b5f-75eb-45f5-83b8-47ae2508461e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62214) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 586.317881] env[62214]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5f3b4b5f-75eb-45f5-83b8-47ae2508461e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 586.318167] env[62214]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5f3b4b5f-75eb-45f5-83b8-47ae2508461e None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=62214) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 588.326180] env[62214]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5f3b4b5f-75eb-45f5-83b8-47ae2508461e None None] Session token - acquiring [ 588.326472] env[62214]: DEBUG networking_nsxv3.common.locking [None req-5f3b4b5f-75eb-45f5-83b8-47ae2508461e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62214) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 588.326659] env[62214]: DEBUG oslo_concurrency.lockutils [None req-5f3b4b5f-75eb-45f5-83b8-47ae2508461e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62214) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 588.326781] env[62214]: DEBUG oslo_concurrency.lockutils [None req-5f3b4b5f-75eb-45f5-83b8-47ae2508461e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62214) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 588.327052] env[62214]: DEBUG oslo_concurrency.lockutils [None req-5f3b4b5f-75eb-45f5-83b8-47ae2508461e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62214) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 588.336483] env[62214]: DEBUG oslo_concurrency.lockutils [None req-5f3b4b5f-75eb-45f5-83b8-47ae2508461e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62214) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 588.336699] env[62214]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5f3b4b5f-75eb-45f5-83b8-47ae2508461e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 588.336838] env[62214]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5f3b4b5f-75eb-45f5-83b8-47ae2508461e None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=62214) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 590.338962] env[62214]: CRITICAL neutron [None req-5f3b4b5f-75eb-45f5-83b8-47ae2508461e None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 590.338962] env[62214]: ERROR neutron Traceback (most recent call last): [ 590.338962] env[62214]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 590.338962] env[62214]: ERROR neutron sys.exit(main()) [ 590.338962] env[62214]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 590.338962] env[62214]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 590.338962] env[62214]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 590.338962] env[62214]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 590.338962] env[62214]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 590.338962] env[62214]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 590.338962] env[62214]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 590.338962] env[62214]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 590.338962] env[62214]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 590.338962] env[62214]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 590.338962] env[62214]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 590.338962] env[62214]: ERROR neutron response = self.get(path=path, params=params) [ 590.339627] env[62214]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 590.339627] env[62214]: ERROR neutron raise RuntimeError(msg, last_err) [ 590.339627] env[62214]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 590.339627] env[62214]: ERROR neutron [ 593.039180] env[62716]: INFO neutron.common.config [-] Logging enabled! [ 593.039512] env[62716]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 593.039708] env[62716]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=62716) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 593.039953] env[62716]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 593.040064] env[62716]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 593.040977] env[62716]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-08cdcef1-7a44-4a88-b7b7-647a41fcab85 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 593.054632] env[62716]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-08cdcef1-7a44-4a88-b7b7-647a41fcab85 None None] Session token - acquiring [ 593.055422] env[62716]: DEBUG networking_nsxv3.common.locking [None req-08cdcef1-7a44-4a88-b7b7-647a41fcab85 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62716) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 593.055625] env[62716]: DEBUG oslo_concurrency.lockutils [None req-08cdcef1-7a44-4a88-b7b7-647a41fcab85 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62716) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 593.055750] env[62716]: DEBUG oslo_concurrency.lockutils [None req-08cdcef1-7a44-4a88-b7b7-647a41fcab85 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62716) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 593.056122] env[62716]: DEBUG oslo_concurrency.lockutils [None req-08cdcef1-7a44-4a88-b7b7-647a41fcab85 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62716) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 593.065781] env[62716]: DEBUG oslo_concurrency.lockutils [None req-08cdcef1-7a44-4a88-b7b7-647a41fcab85 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62716) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 593.066025] env[62716]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-08cdcef1-7a44-4a88-b7b7-647a41fcab85 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 593.066194] env[62716]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-08cdcef1-7a44-4a88-b7b7-647a41fcab85 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=62716) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 595.074125] env[62716]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-08cdcef1-7a44-4a88-b7b7-647a41fcab85 None None] Session token - acquiring [ 595.074362] env[62716]: DEBUG networking_nsxv3.common.locking [None req-08cdcef1-7a44-4a88-b7b7-647a41fcab85 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=62716) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 595.074562] env[62716]: DEBUG oslo_concurrency.lockutils [None req-08cdcef1-7a44-4a88-b7b7-647a41fcab85 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62716) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 595.074615] env[62716]: DEBUG oslo_concurrency.lockutils [None req-08cdcef1-7a44-4a88-b7b7-647a41fcab85 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62716) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 595.077831] env[62716]: DEBUG oslo_concurrency.lockutils [None req-08cdcef1-7a44-4a88-b7b7-647a41fcab85 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62716) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 595.085888] env[62716]: DEBUG oslo_concurrency.lockutils [None req-08cdcef1-7a44-4a88-b7b7-647a41fcab85 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=62716) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 595.085888] env[62716]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-08cdcef1-7a44-4a88-b7b7-647a41fcab85 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 595.085888] env[62716]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-08cdcef1-7a44-4a88-b7b7-647a41fcab85 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=62716) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 597.088580] env[62716]: CRITICAL neutron [None req-08cdcef1-7a44-4a88-b7b7-647a41fcab85 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 597.088580] env[62716]: ERROR neutron Traceback (most recent call last): [ 597.088580] env[62716]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 597.088580] env[62716]: ERROR neutron sys.exit(main()) [ 597.088580] env[62716]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 597.088580] env[62716]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 597.088580] env[62716]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 597.088580] env[62716]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 597.088580] env[62716]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 597.088580] env[62716]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 597.088580] env[62716]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 597.088580] env[62716]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 597.088580] env[62716]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 597.088580] env[62716]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 597.088580] env[62716]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 597.088580] env[62716]: ERROR neutron response = self.get(path=path, params=params) [ 597.089229] env[62716]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 597.089229] env[62716]: ERROR neutron raise RuntimeError(msg, last_err) [ 597.089229] env[62716]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 597.089229] env[62716]: ERROR neutron [ 600.054455] env[63151]: INFO neutron.common.config [-] Logging enabled! [ 600.054731] env[63151]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 600.054958] env[63151]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=63151) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 600.055191] env[63151]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 600.055281] env[63151]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 600.056183] env[63151]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-2c2e4bba-c78b-4225-af0c-24f74206dfd6 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 600.070640] env[63151]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2c2e4bba-c78b-4225-af0c-24f74206dfd6 None None] Session token - acquiring [ 600.071534] env[63151]: DEBUG networking_nsxv3.common.locking [None req-2c2e4bba-c78b-4225-af0c-24f74206dfd6 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=63151) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 600.071711] env[63151]: DEBUG oslo_concurrency.lockutils [None req-2c2e4bba-c78b-4225-af0c-24f74206dfd6 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63151) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 600.072011] env[63151]: DEBUG oslo_concurrency.lockutils [None req-2c2e4bba-c78b-4225-af0c-24f74206dfd6 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63151) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 600.072271] env[63151]: DEBUG oslo_concurrency.lockutils [None req-2c2e4bba-c78b-4225-af0c-24f74206dfd6 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63151) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 600.082302] env[63151]: DEBUG oslo_concurrency.lockutils [None req-2c2e4bba-c78b-4225-af0c-24f74206dfd6 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63151) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 600.082667] env[63151]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2c2e4bba-c78b-4225-af0c-24f74206dfd6 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 600.082819] env[63151]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2c2e4bba-c78b-4225-af0c-24f74206dfd6 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=63151) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 602.091137] env[63151]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2c2e4bba-c78b-4225-af0c-24f74206dfd6 None None] Session token - acquiring [ 602.091458] env[63151]: DEBUG networking_nsxv3.common.locking [None req-2c2e4bba-c78b-4225-af0c-24f74206dfd6 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=63151) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 602.091651] env[63151]: DEBUG oslo_concurrency.lockutils [None req-2c2e4bba-c78b-4225-af0c-24f74206dfd6 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63151) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 602.091773] env[63151]: DEBUG oslo_concurrency.lockutils [None req-2c2e4bba-c78b-4225-af0c-24f74206dfd6 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63151) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 602.092164] env[63151]: DEBUG oslo_concurrency.lockutils [None req-2c2e4bba-c78b-4225-af0c-24f74206dfd6 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63151) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 602.106201] env[63151]: DEBUG oslo_concurrency.lockutils [None req-2c2e4bba-c78b-4225-af0c-24f74206dfd6 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63151) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 602.106201] env[63151]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2c2e4bba-c78b-4225-af0c-24f74206dfd6 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 602.106201] env[63151]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2c2e4bba-c78b-4225-af0c-24f74206dfd6 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=63151) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 604.108983] env[63151]: CRITICAL neutron [None req-2c2e4bba-c78b-4225-af0c-24f74206dfd6 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 604.108983] env[63151]: ERROR neutron Traceback (most recent call last): [ 604.108983] env[63151]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 604.108983] env[63151]: ERROR neutron sys.exit(main()) [ 604.108983] env[63151]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 604.108983] env[63151]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 604.108983] env[63151]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 604.108983] env[63151]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 604.108983] env[63151]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 604.108983] env[63151]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 604.108983] env[63151]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 604.108983] env[63151]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 604.108983] env[63151]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 604.108983] env[63151]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 604.108983] env[63151]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 604.108983] env[63151]: ERROR neutron response = self.get(path=path, params=params) [ 604.110149] env[63151]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 604.110149] env[63151]: ERROR neutron raise RuntimeError(msg, last_err) [ 604.110149] env[63151]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 604.110149] env[63151]: ERROR neutron [ 606.878298] env[63522]: INFO neutron.common.config [-] Logging enabled! [ 606.878599] env[63522]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 606.878797] env[63522]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=63522) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 606.879036] env[63522]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 606.879130] env[63522]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 606.880026] env[63522]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-bdf73491-afa4-48c8-92f1-32f2375b62fb None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 606.893763] env[63522]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bdf73491-afa4-48c8-92f1-32f2375b62fb None None] Session token - acquiring [ 606.894701] env[63522]: DEBUG networking_nsxv3.common.locking [None req-bdf73491-afa4-48c8-92f1-32f2375b62fb None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=63522) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 606.894867] env[63522]: DEBUG oslo_concurrency.lockutils [None req-bdf73491-afa4-48c8-92f1-32f2375b62fb None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63522) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 606.894932] env[63522]: DEBUG oslo_concurrency.lockutils [None req-bdf73491-afa4-48c8-92f1-32f2375b62fb None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63522) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 606.895246] env[63522]: DEBUG oslo_concurrency.lockutils [None req-bdf73491-afa4-48c8-92f1-32f2375b62fb None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63522) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 606.904632] env[63522]: DEBUG oslo_concurrency.lockutils [None req-bdf73491-afa4-48c8-92f1-32f2375b62fb None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63522) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 606.904912] env[63522]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bdf73491-afa4-48c8-92f1-32f2375b62fb None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 606.905067] env[63522]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bdf73491-afa4-48c8-92f1-32f2375b62fb None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=63522) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 608.912978] env[63522]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bdf73491-afa4-48c8-92f1-32f2375b62fb None None] Session token - acquiring [ 608.913299] env[63522]: DEBUG networking_nsxv3.common.locking [None req-bdf73491-afa4-48c8-92f1-32f2375b62fb None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=63522) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 608.913463] env[63522]: DEBUG oslo_concurrency.lockutils [None req-bdf73491-afa4-48c8-92f1-32f2375b62fb None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63522) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 608.913583] env[63522]: DEBUG oslo_concurrency.lockutils [None req-bdf73491-afa4-48c8-92f1-32f2375b62fb None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63522) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 608.913852] env[63522]: DEBUG oslo_concurrency.lockutils [None req-bdf73491-afa4-48c8-92f1-32f2375b62fb None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63522) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 608.923514] env[63522]: DEBUG oslo_concurrency.lockutils [None req-bdf73491-afa4-48c8-92f1-32f2375b62fb None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63522) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 608.923784] env[63522]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bdf73491-afa4-48c8-92f1-32f2375b62fb None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 608.923909] env[63522]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-bdf73491-afa4-48c8-92f1-32f2375b62fb None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=63522) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 610.924913] env[63522]: CRITICAL neutron [None req-bdf73491-afa4-48c8-92f1-32f2375b62fb None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 610.924913] env[63522]: ERROR neutron Traceback (most recent call last): [ 610.924913] env[63522]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 610.924913] env[63522]: ERROR neutron sys.exit(main()) [ 610.924913] env[63522]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 610.924913] env[63522]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 610.924913] env[63522]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 610.924913] env[63522]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 610.924913] env[63522]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 610.924913] env[63522]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 610.924913] env[63522]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 610.924913] env[63522]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 610.924913] env[63522]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 610.924913] env[63522]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 610.924913] env[63522]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 610.924913] env[63522]: ERROR neutron response = self.get(path=path, params=params) [ 610.925520] env[63522]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 610.925520] env[63522]: ERROR neutron raise RuntimeError(msg, last_err) [ 610.925520] env[63522]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 610.925520] env[63522]: ERROR neutron [ 613.680954] env[63998]: INFO neutron.common.config [-] Logging enabled! [ 613.681232] env[63998]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 613.681462] env[63998]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=63998) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 613.681721] env[63998]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 613.681816] env[63998]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 613.682701] env[63998]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-6533420e-424a-441d-aafa-8ded57ed2462 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 613.696509] env[63998]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6533420e-424a-441d-aafa-8ded57ed2462 None None] Session token - acquiring [ 613.697294] env[63998]: DEBUG networking_nsxv3.common.locking [None req-6533420e-424a-441d-aafa-8ded57ed2462 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=63998) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 613.697505] env[63998]: DEBUG oslo_concurrency.lockutils [None req-6533420e-424a-441d-aafa-8ded57ed2462 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63998) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 613.697635] env[63998]: DEBUG oslo_concurrency.lockutils [None req-6533420e-424a-441d-aafa-8ded57ed2462 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63998) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 613.697993] env[63998]: DEBUG oslo_concurrency.lockutils [None req-6533420e-424a-441d-aafa-8ded57ed2462 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63998) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 613.707105] env[63998]: DEBUG oslo_concurrency.lockutils [None req-6533420e-424a-441d-aafa-8ded57ed2462 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63998) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 613.707385] env[63998]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6533420e-424a-441d-aafa-8ded57ed2462 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 613.707536] env[63998]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6533420e-424a-441d-aafa-8ded57ed2462 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=63998) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 615.714414] env[63998]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6533420e-424a-441d-aafa-8ded57ed2462 None None] Session token - acquiring [ 615.714414] env[63998]: DEBUG networking_nsxv3.common.locking [None req-6533420e-424a-441d-aafa-8ded57ed2462 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=63998) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 615.714844] env[63998]: DEBUG oslo_concurrency.lockutils [None req-6533420e-424a-441d-aafa-8ded57ed2462 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63998) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 615.714844] env[63998]: DEBUG oslo_concurrency.lockutils [None req-6533420e-424a-441d-aafa-8ded57ed2462 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63998) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 615.714920] env[63998]: DEBUG oslo_concurrency.lockutils [None req-6533420e-424a-441d-aafa-8ded57ed2462 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63998) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 615.724611] env[63998]: DEBUG oslo_concurrency.lockutils [None req-6533420e-424a-441d-aafa-8ded57ed2462 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=63998) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 615.724948] env[63998]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6533420e-424a-441d-aafa-8ded57ed2462 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 615.725078] env[63998]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-6533420e-424a-441d-aafa-8ded57ed2462 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=63998) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 617.728082] env[63998]: CRITICAL neutron [None req-6533420e-424a-441d-aafa-8ded57ed2462 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 617.728082] env[63998]: ERROR neutron Traceback (most recent call last): [ 617.728082] env[63998]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 617.728082] env[63998]: ERROR neutron sys.exit(main()) [ 617.728082] env[63998]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 617.728082] env[63998]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 617.728082] env[63998]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 617.728082] env[63998]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 617.728082] env[63998]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 617.728082] env[63998]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 617.728082] env[63998]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 617.728082] env[63998]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 617.728082] env[63998]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 617.728082] env[63998]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 617.728082] env[63998]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 617.728082] env[63998]: ERROR neutron response = self.get(path=path, params=params) [ 617.728789] env[63998]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 617.728789] env[63998]: ERROR neutron raise RuntimeError(msg, last_err) [ 617.728789] env[63998]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 617.728789] env[63998]: ERROR neutron [ 620.521904] env[64441]: INFO neutron.common.config [-] Logging enabled! [ 620.522276] env[64441]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 620.522414] env[64441]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=64441) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 620.522663] env[64441]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 620.522761] env[64441]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 620.523654] env[64441]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-77dae345-6fb0-4090-a732-6d7971bc6ad9 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 620.538059] env[64441]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-77dae345-6fb0-4090-a732-6d7971bc6ad9 None None] Session token - acquiring [ 620.538881] env[64441]: DEBUG networking_nsxv3.common.locking [None req-77dae345-6fb0-4090-a732-6d7971bc6ad9 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=64441) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 620.539093] env[64441]: DEBUG oslo_concurrency.lockutils [None req-77dae345-6fb0-4090-a732-6d7971bc6ad9 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64441) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 620.539222] env[64441]: DEBUG oslo_concurrency.lockutils [None req-77dae345-6fb0-4090-a732-6d7971bc6ad9 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64441) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 620.540316] env[64441]: DEBUG oslo_concurrency.lockutils [None req-77dae345-6fb0-4090-a732-6d7971bc6ad9 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64441) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 620.549511] env[64441]: DEBUG oslo_concurrency.lockutils [None req-77dae345-6fb0-4090-a732-6d7971bc6ad9 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64441) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 620.549767] env[64441]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-77dae345-6fb0-4090-a732-6d7971bc6ad9 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 620.549911] env[64441]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-77dae345-6fb0-4090-a732-6d7971bc6ad9 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=64441) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 622.557726] env[64441]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-77dae345-6fb0-4090-a732-6d7971bc6ad9 None None] Session token - acquiring [ 622.557968] env[64441]: DEBUG networking_nsxv3.common.locking [None req-77dae345-6fb0-4090-a732-6d7971bc6ad9 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=64441) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 622.558145] env[64441]: DEBUG oslo_concurrency.lockutils [None req-77dae345-6fb0-4090-a732-6d7971bc6ad9 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64441) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 622.558281] env[64441]: DEBUG oslo_concurrency.lockutils [None req-77dae345-6fb0-4090-a732-6d7971bc6ad9 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64441) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 622.558601] env[64441]: DEBUG oslo_concurrency.lockutils [None req-77dae345-6fb0-4090-a732-6d7971bc6ad9 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64441) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 622.567896] env[64441]: DEBUG oslo_concurrency.lockutils [None req-77dae345-6fb0-4090-a732-6d7971bc6ad9 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64441) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 622.568301] env[64441]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-77dae345-6fb0-4090-a732-6d7971bc6ad9 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 622.568301] env[64441]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-77dae345-6fb0-4090-a732-6d7971bc6ad9 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=64441) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 624.576053] env[64441]: CRITICAL neutron [None req-77dae345-6fb0-4090-a732-6d7971bc6ad9 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 624.576053] env[64441]: ERROR neutron Traceback (most recent call last): [ 624.576053] env[64441]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 624.576053] env[64441]: ERROR neutron sys.exit(main()) [ 624.576053] env[64441]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 624.576053] env[64441]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 624.576053] env[64441]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 624.576053] env[64441]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 624.576053] env[64441]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 624.576053] env[64441]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 624.576053] env[64441]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 624.576053] env[64441]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 624.576053] env[64441]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 624.576053] env[64441]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 624.576053] env[64441]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 624.576053] env[64441]: ERROR neutron response = self.get(path=path, params=params) [ 624.577044] env[64441]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 624.577044] env[64441]: ERROR neutron raise RuntimeError(msg, last_err) [ 624.577044] env[64441]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 624.577044] env[64441]: ERROR neutron [ 627.531464] env[64908]: INFO neutron.common.config [-] Logging enabled! [ 627.531718] env[64908]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 627.532036] env[64908]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=64908) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 627.532350] env[64908]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 627.532350] env[64908]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 627.533223] env[64908]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-f80e5aa5-1994-482d-81f3-99958c3ee1e7 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 627.549129] env[64908]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f80e5aa5-1994-482d-81f3-99958c3ee1e7 None None] Session token - acquiring [ 627.550060] env[64908]: DEBUG networking_nsxv3.common.locking [None req-f80e5aa5-1994-482d-81f3-99958c3ee1e7 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=64908) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 627.550269] env[64908]: DEBUG oslo_concurrency.lockutils [None req-f80e5aa5-1994-482d-81f3-99958c3ee1e7 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64908) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 627.550391] env[64908]: DEBUG oslo_concurrency.lockutils [None req-f80e5aa5-1994-482d-81f3-99958c3ee1e7 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64908) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 627.552056] env[64908]: DEBUG oslo_concurrency.lockutils [None req-f80e5aa5-1994-482d-81f3-99958c3ee1e7 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64908) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 627.561969] env[64908]: DEBUG oslo_concurrency.lockutils [None req-f80e5aa5-1994-482d-81f3-99958c3ee1e7 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64908) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 627.562101] env[64908]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f80e5aa5-1994-482d-81f3-99958c3ee1e7 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 627.562257] env[64908]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f80e5aa5-1994-482d-81f3-99958c3ee1e7 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=64908) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 629.570012] env[64908]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f80e5aa5-1994-482d-81f3-99958c3ee1e7 None None] Session token - acquiring [ 629.570333] env[64908]: DEBUG networking_nsxv3.common.locking [None req-f80e5aa5-1994-482d-81f3-99958c3ee1e7 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=64908) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 629.570521] env[64908]: DEBUG oslo_concurrency.lockutils [None req-f80e5aa5-1994-482d-81f3-99958c3ee1e7 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64908) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 629.570647] env[64908]: DEBUG oslo_concurrency.lockutils [None req-f80e5aa5-1994-482d-81f3-99958c3ee1e7 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64908) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 629.570920] env[64908]: DEBUG oslo_concurrency.lockutils [None req-f80e5aa5-1994-482d-81f3-99958c3ee1e7 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64908) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 629.580981] env[64908]: DEBUG oslo_concurrency.lockutils [None req-f80e5aa5-1994-482d-81f3-99958c3ee1e7 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=64908) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 629.581330] env[64908]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f80e5aa5-1994-482d-81f3-99958c3ee1e7 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 629.581488] env[64908]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-f80e5aa5-1994-482d-81f3-99958c3ee1e7 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=64908) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 631.584648] env[64908]: CRITICAL neutron [None req-f80e5aa5-1994-482d-81f3-99958c3ee1e7 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 631.584648] env[64908]: ERROR neutron Traceback (most recent call last): [ 631.584648] env[64908]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 631.584648] env[64908]: ERROR neutron sys.exit(main()) [ 631.584648] env[64908]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 631.584648] env[64908]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 631.584648] env[64908]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 631.584648] env[64908]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 631.584648] env[64908]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 631.584648] env[64908]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 631.584648] env[64908]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 631.584648] env[64908]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 631.584648] env[64908]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 631.584648] env[64908]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 631.584648] env[64908]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 631.584648] env[64908]: ERROR neutron response = self.get(path=path, params=params) [ 631.585419] env[64908]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 631.585419] env[64908]: ERROR neutron raise RuntimeError(msg, last_err) [ 631.585419] env[64908]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 631.585419] env[64908]: ERROR neutron [ 634.372630] env[65647]: INFO neutron.common.config [-] Logging enabled! [ 634.372876] env[65647]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 634.373194] env[65647]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=65647) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 634.373509] env[65647]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 634.373637] env[65647]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 634.374949] env[65647]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-0f33f4cb-4094-4a0d-865c-4e8b30a87e85 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 634.391967] env[65647]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0f33f4cb-4094-4a0d-865c-4e8b30a87e85 None None] Session token - acquiring [ 634.392909] env[65647]: DEBUG networking_nsxv3.common.locking [None req-0f33f4cb-4094-4a0d-865c-4e8b30a87e85 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=65647) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 634.393151] env[65647]: DEBUG oslo_concurrency.lockutils [None req-0f33f4cb-4094-4a0d-865c-4e8b30a87e85 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65647) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 634.393375] env[65647]: DEBUG oslo_concurrency.lockutils [None req-0f33f4cb-4094-4a0d-865c-4e8b30a87e85 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65647) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 634.393757] env[65647]: DEBUG oslo_concurrency.lockutils [None req-0f33f4cb-4094-4a0d-865c-4e8b30a87e85 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65647) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 634.408233] env[65647]: DEBUG oslo_concurrency.lockutils [None req-0f33f4cb-4094-4a0d-865c-4e8b30a87e85 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65647) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 634.408497] env[65647]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0f33f4cb-4094-4a0d-865c-4e8b30a87e85 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 634.408638] env[65647]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0f33f4cb-4094-4a0d-865c-4e8b30a87e85 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=65647) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 636.416512] env[65647]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0f33f4cb-4094-4a0d-865c-4e8b30a87e85 None None] Session token - acquiring [ 636.416961] env[65647]: DEBUG networking_nsxv3.common.locking [None req-0f33f4cb-4094-4a0d-865c-4e8b30a87e85 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=65647) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 636.417113] env[65647]: DEBUG oslo_concurrency.lockutils [None req-0f33f4cb-4094-4a0d-865c-4e8b30a87e85 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65647) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 636.417113] env[65647]: DEBUG oslo_concurrency.lockutils [None req-0f33f4cb-4094-4a0d-865c-4e8b30a87e85 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65647) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 636.417400] env[65647]: DEBUG oslo_concurrency.lockutils [None req-0f33f4cb-4094-4a0d-865c-4e8b30a87e85 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65647) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 636.431939] env[65647]: DEBUG oslo_concurrency.lockutils [None req-0f33f4cb-4094-4a0d-865c-4e8b30a87e85 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=65647) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 636.432425] env[65647]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0f33f4cb-4094-4a0d-865c-4e8b30a87e85 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 636.432627] env[65647]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-0f33f4cb-4094-4a0d-865c-4e8b30a87e85 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=65647) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 638.438336] env[65647]: CRITICAL neutron [None req-0f33f4cb-4094-4a0d-865c-4e8b30a87e85 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 638.438336] env[65647]: ERROR neutron Traceback (most recent call last): [ 638.438336] env[65647]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 638.438336] env[65647]: ERROR neutron sys.exit(main()) [ 638.438336] env[65647]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 638.438336] env[65647]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 638.438336] env[65647]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 638.438336] env[65647]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 638.438336] env[65647]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 638.438336] env[65647]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 638.438336] env[65647]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 638.438336] env[65647]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 638.438336] env[65647]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 638.438336] env[65647]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 638.438336] env[65647]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 638.438336] env[65647]: ERROR neutron response = self.get(path=path, params=params) [ 638.439420] env[65647]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 638.439420] env[65647]: ERROR neutron raise RuntimeError(msg, last_err) [ 638.439420] env[65647]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 638.439420] env[65647]: ERROR neutron [ 641.294847] env[66120]: INFO neutron.common.config [-] Logging enabled! [ 641.295246] env[66120]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 641.295350] env[66120]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=66120) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 641.295704] env[66120]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 641.295795] env[66120]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 641.296760] env[66120]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-ba66b6ba-d703-4786-ac7a-d951b0d3b770 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 641.313235] env[66120]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ba66b6ba-d703-4786-ac7a-d951b0d3b770 None None] Session token - acquiring [ 641.314081] env[66120]: DEBUG networking_nsxv3.common.locking [None req-ba66b6ba-d703-4786-ac7a-d951b0d3b770 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=66120) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 641.314503] env[66120]: DEBUG oslo_concurrency.lockutils [None req-ba66b6ba-d703-4786-ac7a-d951b0d3b770 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66120) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 641.314503] env[66120]: DEBUG oslo_concurrency.lockutils [None req-ba66b6ba-d703-4786-ac7a-d951b0d3b770 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66120) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 641.318027] env[66120]: DEBUG oslo_concurrency.lockutils [None req-ba66b6ba-d703-4786-ac7a-d951b0d3b770 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66120) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 641.327140] env[66120]: DEBUG oslo_concurrency.lockutils [None req-ba66b6ba-d703-4786-ac7a-d951b0d3b770 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66120) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 641.327735] env[66120]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ba66b6ba-d703-4786-ac7a-d951b0d3b770 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 641.330719] env[66120]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ba66b6ba-d703-4786-ac7a-d951b0d3b770 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=66120) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 643.338563] env[66120]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ba66b6ba-d703-4786-ac7a-d951b0d3b770 None None] Session token - acquiring [ 643.338922] env[66120]: DEBUG networking_nsxv3.common.locking [None req-ba66b6ba-d703-4786-ac7a-d951b0d3b770 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=66120) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 643.338922] env[66120]: DEBUG oslo_concurrency.lockutils [None req-ba66b6ba-d703-4786-ac7a-d951b0d3b770 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66120) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 643.339022] env[66120]: DEBUG oslo_concurrency.lockutils [None req-ba66b6ba-d703-4786-ac7a-d951b0d3b770 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66120) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 643.339216] env[66120]: DEBUG oslo_concurrency.lockutils [None req-ba66b6ba-d703-4786-ac7a-d951b0d3b770 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66120) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 643.349324] env[66120]: DEBUG oslo_concurrency.lockutils [None req-ba66b6ba-d703-4786-ac7a-d951b0d3b770 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66120) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 643.349650] env[66120]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ba66b6ba-d703-4786-ac7a-d951b0d3b770 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 643.349788] env[66120]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ba66b6ba-d703-4786-ac7a-d951b0d3b770 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=66120) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 645.356421] env[66120]: CRITICAL neutron [None req-ba66b6ba-d703-4786-ac7a-d951b0d3b770 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 645.356421] env[66120]: ERROR neutron Traceback (most recent call last): [ 645.356421] env[66120]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 645.356421] env[66120]: ERROR neutron sys.exit(main()) [ 645.356421] env[66120]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 645.356421] env[66120]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 645.356421] env[66120]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 645.356421] env[66120]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 645.356421] env[66120]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 645.356421] env[66120]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 645.356421] env[66120]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 645.356421] env[66120]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 645.356421] env[66120]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 645.356421] env[66120]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 645.356421] env[66120]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 645.356421] env[66120]: ERROR neutron response = self.get(path=path, params=params) [ 645.357034] env[66120]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 645.357034] env[66120]: ERROR neutron raise RuntimeError(msg, last_err) [ 645.357034] env[66120]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 645.357034] env[66120]: ERROR neutron [ 648.284749] env[66853]: INFO neutron.common.config [-] Logging enabled! [ 648.285023] env[66853]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 648.285263] env[66853]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=66853) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 648.285503] env[66853]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 648.285598] env[66853]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 648.286487] env[66853]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-9108ae62-9172-453d-ae80-97b7938405c2 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 648.300464] env[66853]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9108ae62-9172-453d-ae80-97b7938405c2 None None] Session token - acquiring [ 648.301286] env[66853]: DEBUG networking_nsxv3.common.locking [None req-9108ae62-9172-453d-ae80-97b7938405c2 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=66853) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 648.301527] env[66853]: DEBUG oslo_concurrency.lockutils [None req-9108ae62-9172-453d-ae80-97b7938405c2 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66853) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 648.301626] env[66853]: DEBUG oslo_concurrency.lockutils [None req-9108ae62-9172-453d-ae80-97b7938405c2 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66853) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 648.301992] env[66853]: DEBUG oslo_concurrency.lockutils [None req-9108ae62-9172-453d-ae80-97b7938405c2 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66853) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 648.312135] env[66853]: DEBUG oslo_concurrency.lockutils [None req-9108ae62-9172-453d-ae80-97b7938405c2 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66853) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 648.312217] env[66853]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9108ae62-9172-453d-ae80-97b7938405c2 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 648.312323] env[66853]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9108ae62-9172-453d-ae80-97b7938405c2 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=66853) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 650.320076] env[66853]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9108ae62-9172-453d-ae80-97b7938405c2 None None] Session token - acquiring [ 650.320434] env[66853]: DEBUG networking_nsxv3.common.locking [None req-9108ae62-9172-453d-ae80-97b7938405c2 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=66853) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 650.320588] env[66853]: DEBUG oslo_concurrency.lockutils [None req-9108ae62-9172-453d-ae80-97b7938405c2 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66853) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 650.320714] env[66853]: DEBUG oslo_concurrency.lockutils [None req-9108ae62-9172-453d-ae80-97b7938405c2 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66853) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 650.320983] env[66853]: DEBUG oslo_concurrency.lockutils [None req-9108ae62-9172-453d-ae80-97b7938405c2 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66853) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 650.330668] env[66853]: DEBUG oslo_concurrency.lockutils [None req-9108ae62-9172-453d-ae80-97b7938405c2 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=66853) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 650.330969] env[66853]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9108ae62-9172-453d-ae80-97b7938405c2 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 650.331280] env[66853]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-9108ae62-9172-453d-ae80-97b7938405c2 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=66853) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 652.332853] env[66853]: CRITICAL neutron [None req-9108ae62-9172-453d-ae80-97b7938405c2 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 652.332853] env[66853]: ERROR neutron Traceback (most recent call last): [ 652.332853] env[66853]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 652.332853] env[66853]: ERROR neutron sys.exit(main()) [ 652.332853] env[66853]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 652.332853] env[66853]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 652.332853] env[66853]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 652.332853] env[66853]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 652.332853] env[66853]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 652.332853] env[66853]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 652.332853] env[66853]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 652.332853] env[66853]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 652.332853] env[66853]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 652.332853] env[66853]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 652.332853] env[66853]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 652.332853] env[66853]: ERROR neutron response = self.get(path=path, params=params) [ 652.333560] env[66853]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 652.333560] env[66853]: ERROR neutron raise RuntimeError(msg, last_err) [ 652.333560] env[66853]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 652.333560] env[66853]: ERROR neutron [ 655.229547] env[67541]: INFO neutron.common.config [-] Logging enabled! [ 655.230052] env[67541]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 655.230919] env[67541]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=67541) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 655.231448] env[67541]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 655.231706] env[67541]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 655.232721] env[67541]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-e6a06746-d14f-4a39-a4f4-cd40462015d5 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 655.247115] env[67541]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e6a06746-d14f-4a39-a4f4-cd40462015d5 None None] Session token - acquiring [ 655.248306] env[67541]: DEBUG networking_nsxv3.common.locking [None req-e6a06746-d14f-4a39-a4f4-cd40462015d5 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=67541) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 655.248707] env[67541]: DEBUG oslo_concurrency.lockutils [None req-e6a06746-d14f-4a39-a4f4-cd40462015d5 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67541) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 655.248942] env[67541]: DEBUG oslo_concurrency.lockutils [None req-e6a06746-d14f-4a39-a4f4-cd40462015d5 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67541) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 655.249434] env[67541]: DEBUG oslo_concurrency.lockutils [None req-e6a06746-d14f-4a39-a4f4-cd40462015d5 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67541) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 655.259931] env[67541]: DEBUG oslo_concurrency.lockutils [None req-e6a06746-d14f-4a39-a4f4-cd40462015d5 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67541) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 655.260431] env[67541]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e6a06746-d14f-4a39-a4f4-cd40462015d5 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 655.260769] env[67541]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e6a06746-d14f-4a39-a4f4-cd40462015d5 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=67541) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 657.268898] env[67541]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e6a06746-d14f-4a39-a4f4-cd40462015d5 None None] Session token - acquiring [ 657.269209] env[67541]: DEBUG networking_nsxv3.common.locking [None req-e6a06746-d14f-4a39-a4f4-cd40462015d5 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=67541) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 657.269443] env[67541]: DEBUG oslo_concurrency.lockutils [None req-e6a06746-d14f-4a39-a4f4-cd40462015d5 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67541) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 657.269557] env[67541]: DEBUG oslo_concurrency.lockutils [None req-e6a06746-d14f-4a39-a4f4-cd40462015d5 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67541) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 657.269835] env[67541]: DEBUG oslo_concurrency.lockutils [None req-e6a06746-d14f-4a39-a4f4-cd40462015d5 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67541) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 657.282422] env[67541]: DEBUG oslo_concurrency.lockutils [None req-e6a06746-d14f-4a39-a4f4-cd40462015d5 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=67541) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 657.282422] env[67541]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e6a06746-d14f-4a39-a4f4-cd40462015d5 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 657.282422] env[67541]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e6a06746-d14f-4a39-a4f4-cd40462015d5 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=67541) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 659.282985] env[67541]: CRITICAL neutron [None req-e6a06746-d14f-4a39-a4f4-cd40462015d5 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 659.282985] env[67541]: ERROR neutron Traceback (most recent call last): [ 659.282985] env[67541]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 659.282985] env[67541]: ERROR neutron sys.exit(main()) [ 659.282985] env[67541]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 659.282985] env[67541]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 659.282985] env[67541]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 659.282985] env[67541]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 659.282985] env[67541]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 659.282985] env[67541]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 659.282985] env[67541]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 659.282985] env[67541]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 659.282985] env[67541]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 659.282985] env[67541]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 659.282985] env[67541]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 659.282985] env[67541]: ERROR neutron response = self.get(path=path, params=params) [ 659.283599] env[67541]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 659.283599] env[67541]: ERROR neutron raise RuntimeError(msg, last_err) [ 659.283599] env[67541]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 659.283599] env[67541]: ERROR neutron [ 662.202585] env[68235]: INFO neutron.common.config [-] Logging enabled! [ 662.203338] env[68235]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 662.203338] env[68235]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=68235) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 662.203444] env[68235]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 662.205026] env[68235]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 662.205026] env[68235]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-ee720c37-a983-4ab3-b374-7920be019ae0 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 662.225934] env[68235]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ee720c37-a983-4ab3-b374-7920be019ae0 None None] Session token - acquiring [ 662.226781] env[68235]: DEBUG networking_nsxv3.common.locking [None req-ee720c37-a983-4ab3-b374-7920be019ae0 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68235) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 662.227042] env[68235]: DEBUG oslo_concurrency.lockutils [None req-ee720c37-a983-4ab3-b374-7920be019ae0 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68235) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 662.227146] env[68235]: DEBUG oslo_concurrency.lockutils [None req-ee720c37-a983-4ab3-b374-7920be019ae0 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68235) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 662.227511] env[68235]: DEBUG oslo_concurrency.lockutils [None req-ee720c37-a983-4ab3-b374-7920be019ae0 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68235) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 662.244083] env[68235]: DEBUG oslo_concurrency.lockutils [None req-ee720c37-a983-4ab3-b374-7920be019ae0 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68235) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 662.244368] env[68235]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ee720c37-a983-4ab3-b374-7920be019ae0 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 662.244526] env[68235]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ee720c37-a983-4ab3-b374-7920be019ae0 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=68235) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 664.251974] env[68235]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ee720c37-a983-4ab3-b374-7920be019ae0 None None] Session token - acquiring [ 664.252319] env[68235]: DEBUG networking_nsxv3.common.locking [None req-ee720c37-a983-4ab3-b374-7920be019ae0 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68235) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 664.252530] env[68235]: DEBUG oslo_concurrency.lockutils [None req-ee720c37-a983-4ab3-b374-7920be019ae0 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68235) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 664.252675] env[68235]: DEBUG oslo_concurrency.lockutils [None req-ee720c37-a983-4ab3-b374-7920be019ae0 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68235) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 664.252959] env[68235]: DEBUG oslo_concurrency.lockutils [None req-ee720c37-a983-4ab3-b374-7920be019ae0 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68235) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 664.263162] env[68235]: DEBUG oslo_concurrency.lockutils [None req-ee720c37-a983-4ab3-b374-7920be019ae0 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68235) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 664.263514] env[68235]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ee720c37-a983-4ab3-b374-7920be019ae0 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 664.263675] env[68235]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ee720c37-a983-4ab3-b374-7920be019ae0 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=68235) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 666.266103] env[68235]: CRITICAL neutron [None req-ee720c37-a983-4ab3-b374-7920be019ae0 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 666.266103] env[68235]: ERROR neutron Traceback (most recent call last): [ 666.266103] env[68235]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 666.266103] env[68235]: ERROR neutron sys.exit(main()) [ 666.266103] env[68235]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 666.266103] env[68235]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 666.266103] env[68235]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 666.266103] env[68235]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 666.266103] env[68235]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 666.266103] env[68235]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 666.266103] env[68235]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 666.266103] env[68235]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 666.266103] env[68235]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 666.266103] env[68235]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 666.266103] env[68235]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 666.266103] env[68235]: ERROR neutron response = self.get(path=path, params=params) [ 666.266686] env[68235]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 666.266686] env[68235]: ERROR neutron raise RuntimeError(msg, last_err) [ 666.266686] env[68235]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 666.266686] env[68235]: ERROR neutron [ 669.014571] env[68782]: INFO neutron.common.config [-] Logging enabled! [ 669.014822] env[68782]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 669.015092] env[68782]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=68782) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 669.015335] env[68782]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 669.015433] env[68782]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 669.016414] env[68782]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-18465761-d848-4759-a44a-571f839cccde None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 669.030639] env[68782]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-18465761-d848-4759-a44a-571f839cccde None None] Session token - acquiring [ 669.031467] env[68782]: DEBUG networking_nsxv3.common.locking [None req-18465761-d848-4759-a44a-571f839cccde None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68782) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 669.031675] env[68782]: DEBUG oslo_concurrency.lockutils [None req-18465761-d848-4759-a44a-571f839cccde None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68782) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 669.031807] env[68782]: DEBUG oslo_concurrency.lockutils [None req-18465761-d848-4759-a44a-571f839cccde None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68782) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 669.032215] env[68782]: DEBUG oslo_concurrency.lockutils [None req-18465761-d848-4759-a44a-571f839cccde None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68782) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 669.042817] env[68782]: DEBUG oslo_concurrency.lockutils [None req-18465761-d848-4759-a44a-571f839cccde None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68782) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 669.043128] env[68782]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-18465761-d848-4759-a44a-571f839cccde None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 669.043215] env[68782]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-18465761-d848-4759-a44a-571f839cccde None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=68782) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 671.049703] env[68782]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-18465761-d848-4759-a44a-571f839cccde None None] Session token - acquiring [ 671.049994] env[68782]: DEBUG networking_nsxv3.common.locking [None req-18465761-d848-4759-a44a-571f839cccde None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=68782) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 671.050365] env[68782]: DEBUG oslo_concurrency.lockutils [None req-18465761-d848-4759-a44a-571f839cccde None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68782) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 671.050365] env[68782]: DEBUG oslo_concurrency.lockutils [None req-18465761-d848-4759-a44a-571f839cccde None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68782) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 671.050584] env[68782]: DEBUG oslo_concurrency.lockutils [None req-18465761-d848-4759-a44a-571f839cccde None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68782) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 671.060414] env[68782]: DEBUG oslo_concurrency.lockutils [None req-18465761-d848-4759-a44a-571f839cccde None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=68782) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 671.060537] env[68782]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-18465761-d848-4759-a44a-571f839cccde None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 671.060688] env[68782]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-18465761-d848-4759-a44a-571f839cccde None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=68782) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 673.063980] env[68782]: CRITICAL neutron [None req-18465761-d848-4759-a44a-571f839cccde None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 673.063980] env[68782]: ERROR neutron Traceback (most recent call last): [ 673.063980] env[68782]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 673.063980] env[68782]: ERROR neutron sys.exit(main()) [ 673.063980] env[68782]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 673.063980] env[68782]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 673.063980] env[68782]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 673.063980] env[68782]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 673.063980] env[68782]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 673.063980] env[68782]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 673.063980] env[68782]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 673.063980] env[68782]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 673.063980] env[68782]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 673.063980] env[68782]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 673.063980] env[68782]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 673.063980] env[68782]: ERROR neutron response = self.get(path=path, params=params) [ 673.064575] env[68782]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 673.064575] env[68782]: ERROR neutron raise RuntimeError(msg, last_err) [ 673.064575] env[68782]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 673.064575] env[68782]: ERROR neutron [ 675.750471] env[69371]: INFO neutron.common.config [-] Logging enabled! [ 675.750871] env[69371]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 675.750985] env[69371]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=69371) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 675.751227] env[69371]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 675.751334] env[69371]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 675.752255] env[69371]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-abdfbd7e-eab7-4a0e-b321-05a08fc823d7 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 675.766395] env[69371]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-abdfbd7e-eab7-4a0e-b321-05a08fc823d7 None None] Session token - acquiring [ 675.767242] env[69371]: DEBUG networking_nsxv3.common.locking [None req-abdfbd7e-eab7-4a0e-b321-05a08fc823d7 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69371) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 675.767452] env[69371]: DEBUG oslo_concurrency.lockutils [None req-abdfbd7e-eab7-4a0e-b321-05a08fc823d7 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69371) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 675.767579] env[69371]: DEBUG oslo_concurrency.lockutils [None req-abdfbd7e-eab7-4a0e-b321-05a08fc823d7 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69371) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 675.767948] env[69371]: DEBUG oslo_concurrency.lockutils [None req-abdfbd7e-eab7-4a0e-b321-05a08fc823d7 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69371) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 675.777982] env[69371]: DEBUG oslo_concurrency.lockutils [None req-abdfbd7e-eab7-4a0e-b321-05a08fc823d7 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69371) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 675.778305] env[69371]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-abdfbd7e-eab7-4a0e-b321-05a08fc823d7 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 675.778543] env[69371]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-abdfbd7e-eab7-4a0e-b321-05a08fc823d7 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=69371) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 677.785962] env[69371]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-abdfbd7e-eab7-4a0e-b321-05a08fc823d7 None None] Session token - acquiring [ 677.786269] env[69371]: DEBUG networking_nsxv3.common.locking [None req-abdfbd7e-eab7-4a0e-b321-05a08fc823d7 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69371) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 677.786464] env[69371]: DEBUG oslo_concurrency.lockutils [None req-abdfbd7e-eab7-4a0e-b321-05a08fc823d7 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69371) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 677.786595] env[69371]: DEBUG oslo_concurrency.lockutils [None req-abdfbd7e-eab7-4a0e-b321-05a08fc823d7 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69371) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 677.786893] env[69371]: DEBUG oslo_concurrency.lockutils [None req-abdfbd7e-eab7-4a0e-b321-05a08fc823d7 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69371) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 677.797593] env[69371]: DEBUG oslo_concurrency.lockutils [None req-abdfbd7e-eab7-4a0e-b321-05a08fc823d7 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69371) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 677.797854] env[69371]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-abdfbd7e-eab7-4a0e-b321-05a08fc823d7 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 677.798013] env[69371]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-abdfbd7e-eab7-4a0e-b321-05a08fc823d7 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=69371) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 679.803085] env[69371]: CRITICAL neutron [None req-abdfbd7e-eab7-4a0e-b321-05a08fc823d7 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 679.803085] env[69371]: ERROR neutron Traceback (most recent call last): [ 679.803085] env[69371]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 679.803085] env[69371]: ERROR neutron sys.exit(main()) [ 679.803085] env[69371]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 679.803085] env[69371]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 679.803085] env[69371]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 679.803085] env[69371]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 679.803085] env[69371]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 679.803085] env[69371]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 679.803085] env[69371]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 679.803085] env[69371]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 679.803085] env[69371]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 679.803085] env[69371]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 679.803085] env[69371]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 679.803085] env[69371]: ERROR neutron response = self.get(path=path, params=params) [ 679.803673] env[69371]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 679.803673] env[69371]: ERROR neutron raise RuntimeError(msg, last_err) [ 679.803673] env[69371]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 679.803673] env[69371]: ERROR neutron [ 682.450017] env[69784]: INFO neutron.common.config [-] Logging enabled! [ 682.450461] env[69784]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 682.450536] env[69784]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=69784) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 682.450784] env[69784]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 682.450952] env[69784]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 682.451790] env[69784]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-c655082e-45da-413f-b0db-c4ece38b551a None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 682.465516] env[69784]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c655082e-45da-413f-b0db-c4ece38b551a None None] Session token - acquiring [ 682.466318] env[69784]: DEBUG networking_nsxv3.common.locking [None req-c655082e-45da-413f-b0db-c4ece38b551a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69784) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 682.466784] env[69784]: DEBUG oslo_concurrency.lockutils [None req-c655082e-45da-413f-b0db-c4ece38b551a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69784) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 682.466784] env[69784]: DEBUG oslo_concurrency.lockutils [None req-c655082e-45da-413f-b0db-c4ece38b551a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69784) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 682.467036] env[69784]: DEBUG oslo_concurrency.lockutils [None req-c655082e-45da-413f-b0db-c4ece38b551a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69784) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 682.476673] env[69784]: DEBUG oslo_concurrency.lockutils [None req-c655082e-45da-413f-b0db-c4ece38b551a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69784) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 682.476956] env[69784]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c655082e-45da-413f-b0db-c4ece38b551a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 682.477140] env[69784]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c655082e-45da-413f-b0db-c4ece38b551a None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=69784) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 684.483786] env[69784]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c655082e-45da-413f-b0db-c4ece38b551a None None] Session token - acquiring [ 684.484306] env[69784]: DEBUG networking_nsxv3.common.locking [None req-c655082e-45da-413f-b0db-c4ece38b551a None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=69784) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 684.484513] env[69784]: DEBUG oslo_concurrency.lockutils [None req-c655082e-45da-413f-b0db-c4ece38b551a None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69784) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 684.484661] env[69784]: DEBUG oslo_concurrency.lockutils [None req-c655082e-45da-413f-b0db-c4ece38b551a None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69784) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 684.484937] env[69784]: DEBUG oslo_concurrency.lockutils [None req-c655082e-45da-413f-b0db-c4ece38b551a None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69784) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 684.494730] env[69784]: DEBUG oslo_concurrency.lockutils [None req-c655082e-45da-413f-b0db-c4ece38b551a None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=69784) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 684.494991] env[69784]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c655082e-45da-413f-b0db-c4ece38b551a None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 684.495145] env[69784]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-c655082e-45da-413f-b0db-c4ece38b551a None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=69784) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 686.496896] env[69784]: CRITICAL neutron [None req-c655082e-45da-413f-b0db-c4ece38b551a None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 686.496896] env[69784]: ERROR neutron Traceback (most recent call last): [ 686.496896] env[69784]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 686.496896] env[69784]: ERROR neutron sys.exit(main()) [ 686.496896] env[69784]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 686.496896] env[69784]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 686.496896] env[69784]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 686.496896] env[69784]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 686.496896] env[69784]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 686.496896] env[69784]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 686.496896] env[69784]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 686.496896] env[69784]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 686.496896] env[69784]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 686.496896] env[69784]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 686.496896] env[69784]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 686.496896] env[69784]: ERROR neutron response = self.get(path=path, params=params) [ 686.497445] env[69784]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 686.497445] env[69784]: ERROR neutron raise RuntimeError(msg, last_err) [ 686.497445] env[69784]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 686.497445] env[69784]: ERROR neutron [ 689.234630] env[70081]: INFO neutron.common.config [-] Logging enabled! [ 689.234958] env[70081]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 689.235148] env[70081]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=70081) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 689.235397] env[70081]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 689.235498] env[70081]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 689.236375] env[70081]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-e0b86ab5-5a04-4afd-a060-ab6b7c4007ad None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 689.250243] env[70081]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e0b86ab5-5a04-4afd-a060-ab6b7c4007ad None None] Session token - acquiring [ 689.251072] env[70081]: DEBUG networking_nsxv3.common.locking [None req-e0b86ab5-5a04-4afd-a060-ab6b7c4007ad None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70081) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 689.251281] env[70081]: DEBUG oslo_concurrency.lockutils [None req-e0b86ab5-5a04-4afd-a060-ab6b7c4007ad None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70081) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 689.251408] env[70081]: DEBUG oslo_concurrency.lockutils [None req-e0b86ab5-5a04-4afd-a060-ab6b7c4007ad None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70081) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 689.251779] env[70081]: DEBUG oslo_concurrency.lockutils [None req-e0b86ab5-5a04-4afd-a060-ab6b7c4007ad None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70081) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 689.261336] env[70081]: DEBUG oslo_concurrency.lockutils [None req-e0b86ab5-5a04-4afd-a060-ab6b7c4007ad None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70081) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 689.261601] env[70081]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e0b86ab5-5a04-4afd-a060-ab6b7c4007ad None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 689.261867] env[70081]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e0b86ab5-5a04-4afd-a060-ab6b7c4007ad None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=70081) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 691.272102] env[70081]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e0b86ab5-5a04-4afd-a060-ab6b7c4007ad None None] Session token - acquiring [ 691.272102] env[70081]: DEBUG networking_nsxv3.common.locking [None req-e0b86ab5-5a04-4afd-a060-ab6b7c4007ad None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70081) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 691.272429] env[70081]: DEBUG oslo_concurrency.lockutils [None req-e0b86ab5-5a04-4afd-a060-ab6b7c4007ad None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70081) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 691.272429] env[70081]: DEBUG oslo_concurrency.lockutils [None req-e0b86ab5-5a04-4afd-a060-ab6b7c4007ad None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70081) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 691.272503] env[70081]: DEBUG oslo_concurrency.lockutils [None req-e0b86ab5-5a04-4afd-a060-ab6b7c4007ad None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70081) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 691.284905] env[70081]: DEBUG oslo_concurrency.lockutils [None req-e0b86ab5-5a04-4afd-a060-ab6b7c4007ad None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70081) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 691.285204] env[70081]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e0b86ab5-5a04-4afd-a060-ab6b7c4007ad None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 691.285308] env[70081]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-e0b86ab5-5a04-4afd-a060-ab6b7c4007ad None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=70081) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 693.288206] env[70081]: CRITICAL neutron [None req-e0b86ab5-5a04-4afd-a060-ab6b7c4007ad None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 693.288206] env[70081]: ERROR neutron Traceback (most recent call last): [ 693.288206] env[70081]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 693.288206] env[70081]: ERROR neutron sys.exit(main()) [ 693.288206] env[70081]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 693.288206] env[70081]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 693.288206] env[70081]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 693.288206] env[70081]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 693.288206] env[70081]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 693.288206] env[70081]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 693.288206] env[70081]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 693.288206] env[70081]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 693.288206] env[70081]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 693.288206] env[70081]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 693.288206] env[70081]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 693.288206] env[70081]: ERROR neutron response = self.get(path=path, params=params) [ 693.288788] env[70081]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 693.288788] env[70081]: ERROR neutron raise RuntimeError(msg, last_err) [ 693.288788] env[70081]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 693.288788] env[70081]: ERROR neutron [ 695.953407] env[70456]: INFO neutron.common.config [-] Logging enabled! [ 695.953709] env[70456]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 695.953913] env[70456]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=70456) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 695.954146] env[70456]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 695.954241] env[70456]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 695.955124] env[70456]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-b6f95676-e6f2-48d2-a15c-b5f69eb373ee None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 695.969172] env[70456]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b6f95676-e6f2-48d2-a15c-b5f69eb373ee None None] Session token - acquiring [ 695.969967] env[70456]: DEBUG networking_nsxv3.common.locking [None req-b6f95676-e6f2-48d2-a15c-b5f69eb373ee None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70456) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 695.970168] env[70456]: DEBUG oslo_concurrency.lockutils [None req-b6f95676-e6f2-48d2-a15c-b5f69eb373ee None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70456) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 695.970300] env[70456]: DEBUG oslo_concurrency.lockutils [None req-b6f95676-e6f2-48d2-a15c-b5f69eb373ee None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70456) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 695.970672] env[70456]: DEBUG oslo_concurrency.lockutils [None req-b6f95676-e6f2-48d2-a15c-b5f69eb373ee None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70456) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 695.980294] env[70456]: DEBUG oslo_concurrency.lockutils [None req-b6f95676-e6f2-48d2-a15c-b5f69eb373ee None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70456) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 695.980597] env[70456]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b6f95676-e6f2-48d2-a15c-b5f69eb373ee None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 695.980778] env[70456]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b6f95676-e6f2-48d2-a15c-b5f69eb373ee None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=70456) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 697.988643] env[70456]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b6f95676-e6f2-48d2-a15c-b5f69eb373ee None None] Session token - acquiring [ 697.988930] env[70456]: DEBUG networking_nsxv3.common.locking [None req-b6f95676-e6f2-48d2-a15c-b5f69eb373ee None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70456) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 697.989112] env[70456]: DEBUG oslo_concurrency.lockutils [None req-b6f95676-e6f2-48d2-a15c-b5f69eb373ee None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70456) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 697.989238] env[70456]: DEBUG oslo_concurrency.lockutils [None req-b6f95676-e6f2-48d2-a15c-b5f69eb373ee None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70456) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 697.989550] env[70456]: DEBUG oslo_concurrency.lockutils [None req-b6f95676-e6f2-48d2-a15c-b5f69eb373ee None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70456) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 697.999614] env[70456]: DEBUG oslo_concurrency.lockutils [None req-b6f95676-e6f2-48d2-a15c-b5f69eb373ee None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70456) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 697.999864] env[70456]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b6f95676-e6f2-48d2-a15c-b5f69eb373ee None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 698.000012] env[70456]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-b6f95676-e6f2-48d2-a15c-b5f69eb373ee None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=70456) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 700.002888] env[70456]: CRITICAL neutron [None req-b6f95676-e6f2-48d2-a15c-b5f69eb373ee None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 700.002888] env[70456]: ERROR neutron Traceback (most recent call last): [ 700.002888] env[70456]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 700.002888] env[70456]: ERROR neutron sys.exit(main()) [ 700.002888] env[70456]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 700.002888] env[70456]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 700.002888] env[70456]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 700.002888] env[70456]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 700.002888] env[70456]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 700.002888] env[70456]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 700.002888] env[70456]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 700.002888] env[70456]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 700.002888] env[70456]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 700.002888] env[70456]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 700.002888] env[70456]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 700.002888] env[70456]: ERROR neutron response = self.get(path=path, params=params) [ 700.003676] env[70456]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 700.003676] env[70456]: ERROR neutron raise RuntimeError(msg, last_err) [ 700.003676] env[70456]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 700.003676] env[70456]: ERROR neutron [ 702.687653] env[70773]: INFO neutron.common.config [-] Logging enabled! [ 702.687929] env[70773]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 702.688218] env[70773]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=70773) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 702.688464] env[70773]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 702.688563] env[70773]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 702.689522] env[70773]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-3820ffbd-253c-410b-ad66-9476611c7bc8 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 702.703095] env[70773]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3820ffbd-253c-410b-ad66-9476611c7bc8 None None] Session token - acquiring [ 702.703594] env[70773]: DEBUG networking_nsxv3.common.locking [None req-3820ffbd-253c-410b-ad66-9476611c7bc8 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70773) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 702.703798] env[70773]: DEBUG oslo_concurrency.lockutils [None req-3820ffbd-253c-410b-ad66-9476611c7bc8 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70773) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 702.703927] env[70773]: DEBUG oslo_concurrency.lockutils [None req-3820ffbd-253c-410b-ad66-9476611c7bc8 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70773) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 702.704314] env[70773]: DEBUG oslo_concurrency.lockutils [None req-3820ffbd-253c-410b-ad66-9476611c7bc8 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70773) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 702.714749] env[70773]: DEBUG oslo_concurrency.lockutils [None req-3820ffbd-253c-410b-ad66-9476611c7bc8 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70773) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 702.715043] env[70773]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3820ffbd-253c-410b-ad66-9476611c7bc8 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 702.715200] env[70773]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3820ffbd-253c-410b-ad66-9476611c7bc8 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=70773) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 704.721806] env[70773]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3820ffbd-253c-410b-ad66-9476611c7bc8 None None] Session token - acquiring [ 704.722108] env[70773]: DEBUG networking_nsxv3.common.locking [None req-3820ffbd-253c-410b-ad66-9476611c7bc8 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=70773) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 704.722298] env[70773]: DEBUG oslo_concurrency.lockutils [None req-3820ffbd-253c-410b-ad66-9476611c7bc8 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70773) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 704.722430] env[70773]: DEBUG oslo_concurrency.lockutils [None req-3820ffbd-253c-410b-ad66-9476611c7bc8 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70773) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 704.722715] env[70773]: DEBUG oslo_concurrency.lockutils [None req-3820ffbd-253c-410b-ad66-9476611c7bc8 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70773) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 704.734424] env[70773]: DEBUG oslo_concurrency.lockutils [None req-3820ffbd-253c-410b-ad66-9476611c7bc8 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=70773) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 704.734707] env[70773]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3820ffbd-253c-410b-ad66-9476611c7bc8 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 704.734853] env[70773]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-3820ffbd-253c-410b-ad66-9476611c7bc8 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=70773) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 706.736862] env[70773]: CRITICAL neutron [None req-3820ffbd-253c-410b-ad66-9476611c7bc8 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 706.736862] env[70773]: ERROR neutron Traceback (most recent call last): [ 706.736862] env[70773]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 706.736862] env[70773]: ERROR neutron sys.exit(main()) [ 706.736862] env[70773]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 706.736862] env[70773]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 706.736862] env[70773]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 706.736862] env[70773]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 706.736862] env[70773]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 706.736862] env[70773]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 706.736862] env[70773]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 706.736862] env[70773]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 706.736862] env[70773]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 706.736862] env[70773]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 706.736862] env[70773]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 706.736862] env[70773]: ERROR neutron response = self.get(path=path, params=params) [ 706.737565] env[70773]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 706.737565] env[70773]: ERROR neutron raise RuntimeError(msg, last_err) [ 706.737565] env[70773]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 706.737565] env[70773]: ERROR neutron [ 709.659033] env[71317]: INFO neutron.common.config [-] Logging enabled! [ 709.659306] env[71317]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 709.659664] env[71317]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=71317) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 709.660415] env[71317]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 709.660415] env[71317]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 709.660995] env[71317]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-1b227580-cb81-4fb1-b0dd-245baec053f7 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 709.675062] env[71317]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1b227580-cb81-4fb1-b0dd-245baec053f7 None None] Session token - acquiring [ 709.675899] env[71317]: DEBUG networking_nsxv3.common.locking [None req-1b227580-cb81-4fb1-b0dd-245baec053f7 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71317) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 709.676131] env[71317]: DEBUG oslo_concurrency.lockutils [None req-1b227580-cb81-4fb1-b0dd-245baec053f7 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71317) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 709.676270] env[71317]: DEBUG oslo_concurrency.lockutils [None req-1b227580-cb81-4fb1-b0dd-245baec053f7 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71317) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 709.676631] env[71317]: DEBUG oslo_concurrency.lockutils [None req-1b227580-cb81-4fb1-b0dd-245baec053f7 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71317) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 709.692501] env[71317]: DEBUG oslo_concurrency.lockutils [None req-1b227580-cb81-4fb1-b0dd-245baec053f7 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71317) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 709.692501] env[71317]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1b227580-cb81-4fb1-b0dd-245baec053f7 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 709.692501] env[71317]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1b227580-cb81-4fb1-b0dd-245baec053f7 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=71317) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 711.697803] env[71317]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1b227580-cb81-4fb1-b0dd-245baec053f7 None None] Session token - acquiring [ 711.698141] env[71317]: DEBUG networking_nsxv3.common.locking [None req-1b227580-cb81-4fb1-b0dd-245baec053f7 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71317) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 711.698292] env[71317]: DEBUG oslo_concurrency.lockutils [None req-1b227580-cb81-4fb1-b0dd-245baec053f7 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71317) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 711.698422] env[71317]: DEBUG oslo_concurrency.lockutils [None req-1b227580-cb81-4fb1-b0dd-245baec053f7 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71317) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 711.698711] env[71317]: DEBUG oslo_concurrency.lockutils [None req-1b227580-cb81-4fb1-b0dd-245baec053f7 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71317) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 711.708586] env[71317]: DEBUG oslo_concurrency.lockutils [None req-1b227580-cb81-4fb1-b0dd-245baec053f7 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71317) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 711.708837] env[71317]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1b227580-cb81-4fb1-b0dd-245baec053f7 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 711.708974] env[71317]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-1b227580-cb81-4fb1-b0dd-245baec053f7 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=71317) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 713.711852] env[71317]: CRITICAL neutron [None req-1b227580-cb81-4fb1-b0dd-245baec053f7 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 713.711852] env[71317]: ERROR neutron Traceback (most recent call last): [ 713.711852] env[71317]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 713.711852] env[71317]: ERROR neutron sys.exit(main()) [ 713.711852] env[71317]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 713.711852] env[71317]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 713.711852] env[71317]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 713.711852] env[71317]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 713.711852] env[71317]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 713.711852] env[71317]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 713.711852] env[71317]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 713.711852] env[71317]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 713.711852] env[71317]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 713.711852] env[71317]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 713.711852] env[71317]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 713.711852] env[71317]: ERROR neutron response = self.get(path=path, params=params) [ 713.712520] env[71317]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 713.712520] env[71317]: ERROR neutron raise RuntimeError(msg, last_err) [ 713.712520] env[71317]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 713.712520] env[71317]: ERROR neutron [ 716.504920] env[71802]: INFO neutron.common.config [-] Logging enabled! [ 716.505208] env[71802]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 716.505441] env[71802]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=71802) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 716.505688] env[71802]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 716.505786] env[71802]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 716.506670] env[71802]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-2c083641-dfac-4ffc-8297-f26801d5d8ce None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 716.521347] env[71802]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2c083641-dfac-4ffc-8297-f26801d5d8ce None None] Session token - acquiring [ 716.522174] env[71802]: DEBUG networking_nsxv3.common.locking [None req-2c083641-dfac-4ffc-8297-f26801d5d8ce None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71802) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 716.522602] env[71802]: DEBUG oslo_concurrency.lockutils [None req-2c083641-dfac-4ffc-8297-f26801d5d8ce None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71802) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 716.522602] env[71802]: DEBUG oslo_concurrency.lockutils [None req-2c083641-dfac-4ffc-8297-f26801d5d8ce None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71802) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 716.522874] env[71802]: DEBUG oslo_concurrency.lockutils [None req-2c083641-dfac-4ffc-8297-f26801d5d8ce None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71802) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 716.532698] env[71802]: DEBUG oslo_concurrency.lockutils [None req-2c083641-dfac-4ffc-8297-f26801d5d8ce None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71802) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 716.532980] env[71802]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2c083641-dfac-4ffc-8297-f26801d5d8ce None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 716.533158] env[71802]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2c083641-dfac-4ffc-8297-f26801d5d8ce None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=71802) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 718.540832] env[71802]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2c083641-dfac-4ffc-8297-f26801d5d8ce None None] Session token - acquiring [ 718.541147] env[71802]: DEBUG networking_nsxv3.common.locking [None req-2c083641-dfac-4ffc-8297-f26801d5d8ce None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=71802) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 718.541340] env[71802]: DEBUG oslo_concurrency.lockutils [None req-2c083641-dfac-4ffc-8297-f26801d5d8ce None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71802) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 718.541466] env[71802]: DEBUG oslo_concurrency.lockutils [None req-2c083641-dfac-4ffc-8297-f26801d5d8ce None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71802) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 718.541746] env[71802]: DEBUG oslo_concurrency.lockutils [None req-2c083641-dfac-4ffc-8297-f26801d5d8ce None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71802) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 718.558381] env[71802]: DEBUG oslo_concurrency.lockutils [None req-2c083641-dfac-4ffc-8297-f26801d5d8ce None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=71802) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 718.558381] env[71802]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2c083641-dfac-4ffc-8297-f26801d5d8ce None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 718.558381] env[71802]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-2c083641-dfac-4ffc-8297-f26801d5d8ce None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=71802) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 720.560974] env[71802]: CRITICAL neutron [None req-2c083641-dfac-4ffc-8297-f26801d5d8ce None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 720.560974] env[71802]: ERROR neutron Traceback (most recent call last): [ 720.560974] env[71802]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 720.560974] env[71802]: ERROR neutron sys.exit(main()) [ 720.560974] env[71802]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 720.560974] env[71802]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 720.560974] env[71802]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 720.560974] env[71802]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 720.560974] env[71802]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 720.560974] env[71802]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 720.560974] env[71802]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 720.560974] env[71802]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 720.560974] env[71802]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 720.560974] env[71802]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 720.560974] env[71802]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 720.560974] env[71802]: ERROR neutron response = self.get(path=path, params=params) [ 720.561778] env[71802]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 720.561778] env[71802]: ERROR neutron raise RuntimeError(msg, last_err) [ 720.561778] env[71802]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 720.561778] env[71802]: ERROR neutron [ 723.188887] env[72442]: INFO neutron.common.config [-] Logging enabled! [ 723.188887] env[72442]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 723.188887] env[72442]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=72442) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 723.188887] env[72442]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 723.188887] env[72442]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 723.189464] env[72442]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-525c7605-acaa-4863-8dde-68a3450955f9 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 723.204736] env[72442]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-525c7605-acaa-4863-8dde-68a3450955f9 None None] Session token - acquiring [ 723.205625] env[72442]: DEBUG networking_nsxv3.common.locking [None req-525c7605-acaa-4863-8dde-68a3450955f9 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72442) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 723.205849] env[72442]: DEBUG oslo_concurrency.lockutils [None req-525c7605-acaa-4863-8dde-68a3450955f9 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72442) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 723.206333] env[72442]: DEBUG oslo_concurrency.lockutils [None req-525c7605-acaa-4863-8dde-68a3450955f9 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72442) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 723.206333] env[72442]: DEBUG oslo_concurrency.lockutils [None req-525c7605-acaa-4863-8dde-68a3450955f9 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72442) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 723.216186] env[72442]: DEBUG oslo_concurrency.lockutils [None req-525c7605-acaa-4863-8dde-68a3450955f9 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72442) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 723.216443] env[72442]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-525c7605-acaa-4863-8dde-68a3450955f9 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 723.216632] env[72442]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-525c7605-acaa-4863-8dde-68a3450955f9 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=72442) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 725.224752] env[72442]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-525c7605-acaa-4863-8dde-68a3450955f9 None None] Session token - acquiring [ 725.225069] env[72442]: DEBUG networking_nsxv3.common.locking [None req-525c7605-acaa-4863-8dde-68a3450955f9 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72442) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 725.225243] env[72442]: DEBUG oslo_concurrency.lockutils [None req-525c7605-acaa-4863-8dde-68a3450955f9 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72442) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 725.225363] env[72442]: DEBUG oslo_concurrency.lockutils [None req-525c7605-acaa-4863-8dde-68a3450955f9 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72442) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 725.225651] env[72442]: DEBUG oslo_concurrency.lockutils [None req-525c7605-acaa-4863-8dde-68a3450955f9 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72442) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 725.235924] env[72442]: DEBUG oslo_concurrency.lockutils [None req-525c7605-acaa-4863-8dde-68a3450955f9 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72442) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 725.236573] env[72442]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-525c7605-acaa-4863-8dde-68a3450955f9 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 725.236573] env[72442]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-525c7605-acaa-4863-8dde-68a3450955f9 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=72442) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 727.239264] env[72442]: CRITICAL neutron [None req-525c7605-acaa-4863-8dde-68a3450955f9 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 727.239264] env[72442]: ERROR neutron Traceback (most recent call last): [ 727.239264] env[72442]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 727.239264] env[72442]: ERROR neutron sys.exit(main()) [ 727.239264] env[72442]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 727.239264] env[72442]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 727.239264] env[72442]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 727.239264] env[72442]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 727.239264] env[72442]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 727.239264] env[72442]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 727.239264] env[72442]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 727.239264] env[72442]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 727.239264] env[72442]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 727.239264] env[72442]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 727.239264] env[72442]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 727.239264] env[72442]: ERROR neutron response = self.get(path=path, params=params) [ 727.239945] env[72442]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 727.239945] env[72442]: ERROR neutron raise RuntimeError(msg, last_err) [ 727.239945] env[72442]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 727.239945] env[72442]: ERROR neutron [ 730.012202] env[72997]: INFO neutron.common.config [-] Logging enabled! [ 730.012202] env[72997]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 730.012202] env[72997]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=72997) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 730.012202] env[72997]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 730.012202] env[72997]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 730.013290] env[72997]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-4f45ff6e-eaec-4081-97bf-4aaa7804d155 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 730.034652] env[72997]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4f45ff6e-eaec-4081-97bf-4aaa7804d155 None None] Session token - acquiring [ 730.036293] env[72997]: DEBUG networking_nsxv3.common.locking [None req-4f45ff6e-eaec-4081-97bf-4aaa7804d155 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72997) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 730.036293] env[72997]: DEBUG oslo_concurrency.lockutils [None req-4f45ff6e-eaec-4081-97bf-4aaa7804d155 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72997) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 730.036795] env[72997]: DEBUG oslo_concurrency.lockutils [None req-4f45ff6e-eaec-4081-97bf-4aaa7804d155 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72997) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 730.036795] env[72997]: DEBUG oslo_concurrency.lockutils [None req-4f45ff6e-eaec-4081-97bf-4aaa7804d155 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72997) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 730.048136] env[72997]: DEBUG oslo_concurrency.lockutils [None req-4f45ff6e-eaec-4081-97bf-4aaa7804d155 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72997) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 730.048368] env[72997]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4f45ff6e-eaec-4081-97bf-4aaa7804d155 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 730.048521] env[72997]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4f45ff6e-eaec-4081-97bf-4aaa7804d155 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=72997) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 732.056455] env[72997]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4f45ff6e-eaec-4081-97bf-4aaa7804d155 None None] Session token - acquiring [ 732.056762] env[72997]: DEBUG networking_nsxv3.common.locking [None req-4f45ff6e-eaec-4081-97bf-4aaa7804d155 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=72997) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 732.056961] env[72997]: DEBUG oslo_concurrency.lockutils [None req-4f45ff6e-eaec-4081-97bf-4aaa7804d155 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72997) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 732.057051] env[72997]: DEBUG oslo_concurrency.lockutils [None req-4f45ff6e-eaec-4081-97bf-4aaa7804d155 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72997) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 732.057334] env[72997]: DEBUG oslo_concurrency.lockutils [None req-4f45ff6e-eaec-4081-97bf-4aaa7804d155 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72997) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 732.067196] env[72997]: DEBUG oslo_concurrency.lockutils [None req-4f45ff6e-eaec-4081-97bf-4aaa7804d155 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=72997) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 732.067459] env[72997]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4f45ff6e-eaec-4081-97bf-4aaa7804d155 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 732.067590] env[72997]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4f45ff6e-eaec-4081-97bf-4aaa7804d155 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=72997) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 734.068862] env[72997]: CRITICAL neutron [None req-4f45ff6e-eaec-4081-97bf-4aaa7804d155 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 734.068862] env[72997]: ERROR neutron Traceback (most recent call last): [ 734.068862] env[72997]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 734.068862] env[72997]: ERROR neutron sys.exit(main()) [ 734.068862] env[72997]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 734.068862] env[72997]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 734.068862] env[72997]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 734.068862] env[72997]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 734.068862] env[72997]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 734.068862] env[72997]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 734.068862] env[72997]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 734.068862] env[72997]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 734.068862] env[72997]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 734.068862] env[72997]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 734.068862] env[72997]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 734.068862] env[72997]: ERROR neutron response = self.get(path=path, params=params) [ 734.069631] env[72997]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 734.069631] env[72997]: ERROR neutron raise RuntimeError(msg, last_err) [ 734.069631] env[72997]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 734.069631] env[72997]: ERROR neutron [ 736.696857] env[73467]: INFO neutron.common.config [-] Logging enabled! [ 736.697176] env[73467]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 736.697426] env[73467]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=73467) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 736.697692] env[73467]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 736.697805] env[73467]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 736.698668] env[73467]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-4a1b0f53-d4f8-4b6e-b515-20848ec86bef None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 736.712656] env[73467]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4a1b0f53-d4f8-4b6e-b515-20848ec86bef None None] Session token - acquiring [ 736.713483] env[73467]: DEBUG networking_nsxv3.common.locking [None req-4a1b0f53-d4f8-4b6e-b515-20848ec86bef None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73467) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 736.713702] env[73467]: DEBUG oslo_concurrency.lockutils [None req-4a1b0f53-d4f8-4b6e-b515-20848ec86bef None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73467) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 736.713831] env[73467]: DEBUG oslo_concurrency.lockutils [None req-4a1b0f53-d4f8-4b6e-b515-20848ec86bef None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73467) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 736.714204] env[73467]: DEBUG oslo_concurrency.lockutils [None req-4a1b0f53-d4f8-4b6e-b515-20848ec86bef None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73467) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 736.723703] env[73467]: DEBUG oslo_concurrency.lockutils [None req-4a1b0f53-d4f8-4b6e-b515-20848ec86bef None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73467) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 736.724023] env[73467]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4a1b0f53-d4f8-4b6e-b515-20848ec86bef None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 736.724208] env[73467]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4a1b0f53-d4f8-4b6e-b515-20848ec86bef None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=73467) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 738.732253] env[73467]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4a1b0f53-d4f8-4b6e-b515-20848ec86bef None None] Session token - acquiring [ 738.732586] env[73467]: DEBUG networking_nsxv3.common.locking [None req-4a1b0f53-d4f8-4b6e-b515-20848ec86bef None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=73467) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 738.732868] env[73467]: DEBUG oslo_concurrency.lockutils [None req-4a1b0f53-d4f8-4b6e-b515-20848ec86bef None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73467) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 738.732931] env[73467]: DEBUG oslo_concurrency.lockutils [None req-4a1b0f53-d4f8-4b6e-b515-20848ec86bef None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73467) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 738.733169] env[73467]: DEBUG oslo_concurrency.lockutils [None req-4a1b0f53-d4f8-4b6e-b515-20848ec86bef None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73467) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 738.743194] env[73467]: DEBUG oslo_concurrency.lockutils [None req-4a1b0f53-d4f8-4b6e-b515-20848ec86bef None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=73467) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 738.743448] env[73467]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4a1b0f53-d4f8-4b6e-b515-20848ec86bef None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 738.743590] env[73467]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-4a1b0f53-d4f8-4b6e-b515-20848ec86bef None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=73467) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 740.749528] env[73467]: CRITICAL neutron [None req-4a1b0f53-d4f8-4b6e-b515-20848ec86bef None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 740.749528] env[73467]: ERROR neutron Traceback (most recent call last): [ 740.749528] env[73467]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 740.749528] env[73467]: ERROR neutron sys.exit(main()) [ 740.749528] env[73467]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 740.749528] env[73467]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 740.749528] env[73467]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 740.749528] env[73467]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 740.749528] env[73467]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 740.749528] env[73467]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 740.749528] env[73467]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 740.749528] env[73467]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 740.749528] env[73467]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 740.749528] env[73467]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 740.749528] env[73467]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 740.749528] env[73467]: ERROR neutron response = self.get(path=path, params=params) [ 740.750344] env[73467]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 740.750344] env[73467]: ERROR neutron raise RuntimeError(msg, last_err) [ 740.750344] env[73467]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 740.750344] env[73467]: ERROR neutron [ 743.499819] env[74029]: INFO neutron.common.config [-] Logging enabled! [ 743.499819] env[74029]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 743.500330] env[74029]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74029) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 743.500330] env[74029]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 743.500330] env[74029]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 743.501163] env[74029]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-8f3ae632-85ba-47ce-88a7-1165540d84a9 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 743.518705] env[74029]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8f3ae632-85ba-47ce-88a7-1165540d84a9 None None] Session token - acquiring [ 743.519583] env[74029]: DEBUG networking_nsxv3.common.locking [None req-8f3ae632-85ba-47ce-88a7-1165540d84a9 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74029) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 743.519857] env[74029]: DEBUG oslo_concurrency.lockutils [None req-8f3ae632-85ba-47ce-88a7-1165540d84a9 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74029) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 743.519939] env[74029]: DEBUG oslo_concurrency.lockutils [None req-8f3ae632-85ba-47ce-88a7-1165540d84a9 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74029) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 743.520311] env[74029]: DEBUG oslo_concurrency.lockutils [None req-8f3ae632-85ba-47ce-88a7-1165540d84a9 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74029) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 743.539956] env[74029]: DEBUG oslo_concurrency.lockutils [None req-8f3ae632-85ba-47ce-88a7-1165540d84a9 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74029) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 743.539956] env[74029]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8f3ae632-85ba-47ce-88a7-1165540d84a9 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 743.539956] env[74029]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8f3ae632-85ba-47ce-88a7-1165540d84a9 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74029) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 745.545680] env[74029]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8f3ae632-85ba-47ce-88a7-1165540d84a9 None None] Session token - acquiring [ 745.545948] env[74029]: DEBUG networking_nsxv3.common.locking [None req-8f3ae632-85ba-47ce-88a7-1165540d84a9 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74029) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 745.545948] env[74029]: DEBUG oslo_concurrency.lockutils [None req-8f3ae632-85ba-47ce-88a7-1165540d84a9 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74029) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 745.546043] env[74029]: DEBUG oslo_concurrency.lockutils [None req-8f3ae632-85ba-47ce-88a7-1165540d84a9 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74029) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 745.546629] env[74029]: DEBUG oslo_concurrency.lockutils [None req-8f3ae632-85ba-47ce-88a7-1165540d84a9 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74029) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 745.556303] env[74029]: DEBUG oslo_concurrency.lockutils [None req-8f3ae632-85ba-47ce-88a7-1165540d84a9 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74029) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 745.556620] env[74029]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8f3ae632-85ba-47ce-88a7-1165540d84a9 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 745.556781] env[74029]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8f3ae632-85ba-47ce-88a7-1165540d84a9 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74029) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 747.559702] env[74029]: CRITICAL neutron [None req-8f3ae632-85ba-47ce-88a7-1165540d84a9 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 747.559702] env[74029]: ERROR neutron Traceback (most recent call last): [ 747.559702] env[74029]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 747.559702] env[74029]: ERROR neutron sys.exit(main()) [ 747.559702] env[74029]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 747.559702] env[74029]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 747.559702] env[74029]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 747.559702] env[74029]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 747.559702] env[74029]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 747.559702] env[74029]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 747.559702] env[74029]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 747.559702] env[74029]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 747.559702] env[74029]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 747.559702] env[74029]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 747.559702] env[74029]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 747.559702] env[74029]: ERROR neutron response = self.get(path=path, params=params) [ 747.560405] env[74029]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 747.560405] env[74029]: ERROR neutron raise RuntimeError(msg, last_err) [ 747.560405] env[74029]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 747.560405] env[74029]: ERROR neutron [ 750.263141] env[74464]: INFO neutron.common.config [-] Logging enabled! [ 750.263419] env[74464]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 750.263650] env[74464]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74464) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 750.263886] env[74464]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 750.263976] env[74464]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 750.265012] env[74464]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-5654025c-a653-474b-9f61-16889ca1ba82 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 750.279065] env[74464]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5654025c-a653-474b-9f61-16889ca1ba82 None None] Session token - acquiring [ 750.279956] env[74464]: DEBUG networking_nsxv3.common.locking [None req-5654025c-a653-474b-9f61-16889ca1ba82 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74464) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 750.280224] env[74464]: DEBUG oslo_concurrency.lockutils [None req-5654025c-a653-474b-9f61-16889ca1ba82 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74464) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 750.280346] env[74464]: DEBUG oslo_concurrency.lockutils [None req-5654025c-a653-474b-9f61-16889ca1ba82 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74464) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 750.280746] env[74464]: DEBUG oslo_concurrency.lockutils [None req-5654025c-a653-474b-9f61-16889ca1ba82 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74464) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 750.290512] env[74464]: DEBUG oslo_concurrency.lockutils [None req-5654025c-a653-474b-9f61-16889ca1ba82 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74464) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 750.290817] env[74464]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5654025c-a653-474b-9f61-16889ca1ba82 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 750.290961] env[74464]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5654025c-a653-474b-9f61-16889ca1ba82 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74464) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 752.297793] env[74464]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5654025c-a653-474b-9f61-16889ca1ba82 None None] Session token - acquiring [ 752.298171] env[74464]: DEBUG networking_nsxv3.common.locking [None req-5654025c-a653-474b-9f61-16889ca1ba82 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74464) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 752.298274] env[74464]: DEBUG oslo_concurrency.lockutils [None req-5654025c-a653-474b-9f61-16889ca1ba82 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74464) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 752.298397] env[74464]: DEBUG oslo_concurrency.lockutils [None req-5654025c-a653-474b-9f61-16889ca1ba82 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74464) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 752.298673] env[74464]: DEBUG oslo_concurrency.lockutils [None req-5654025c-a653-474b-9f61-16889ca1ba82 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74464) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 752.308477] env[74464]: DEBUG oslo_concurrency.lockutils [None req-5654025c-a653-474b-9f61-16889ca1ba82 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74464) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 752.308757] env[74464]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5654025c-a653-474b-9f61-16889ca1ba82 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 752.308959] env[74464]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-5654025c-a653-474b-9f61-16889ca1ba82 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74464) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 754.311166] env[74464]: CRITICAL neutron [None req-5654025c-a653-474b-9f61-16889ca1ba82 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 754.311166] env[74464]: ERROR neutron Traceback (most recent call last): [ 754.311166] env[74464]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 754.311166] env[74464]: ERROR neutron sys.exit(main()) [ 754.311166] env[74464]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 754.311166] env[74464]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 754.311166] env[74464]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 754.311166] env[74464]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 754.311166] env[74464]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 754.311166] env[74464]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 754.311166] env[74464]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 754.311166] env[74464]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 754.311166] env[74464]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 754.311166] env[74464]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 754.311166] env[74464]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 754.311166] env[74464]: ERROR neutron response = self.get(path=path, params=params) [ 754.311931] env[74464]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 754.311931] env[74464]: ERROR neutron raise RuntimeError(msg, last_err) [ 754.311931] env[74464]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 754.311931] env[74464]: ERROR neutron [ 757.037951] env[74918]: INFO neutron.common.config [-] Logging enabled! [ 757.038297] env[74918]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 757.038477] env[74918]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=74918) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 757.038710] env[74918]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 757.038801] env[74918]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 757.039717] env[74918]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-63544f80-a4cf-448b-88ca-5027cd09d4b0 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 757.053910] env[74918]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63544f80-a4cf-448b-88ca-5027cd09d4b0 None None] Session token - acquiring [ 757.054752] env[74918]: DEBUG networking_nsxv3.common.locking [None req-63544f80-a4cf-448b-88ca-5027cd09d4b0 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74918) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 757.055019] env[74918]: DEBUG oslo_concurrency.lockutils [None req-63544f80-a4cf-448b-88ca-5027cd09d4b0 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74918) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 757.055083] env[74918]: DEBUG oslo_concurrency.lockutils [None req-63544f80-a4cf-448b-88ca-5027cd09d4b0 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74918) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 757.055449] env[74918]: DEBUG oslo_concurrency.lockutils [None req-63544f80-a4cf-448b-88ca-5027cd09d4b0 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74918) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 757.065157] env[74918]: DEBUG oslo_concurrency.lockutils [None req-63544f80-a4cf-448b-88ca-5027cd09d4b0 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74918) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 757.065473] env[74918]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63544f80-a4cf-448b-88ca-5027cd09d4b0 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 757.065648] env[74918]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63544f80-a4cf-448b-88ca-5027cd09d4b0 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=74918) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 759.071835] env[74918]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63544f80-a4cf-448b-88ca-5027cd09d4b0 None None] Session token - acquiring [ 759.072089] env[74918]: DEBUG networking_nsxv3.common.locking [None req-63544f80-a4cf-448b-88ca-5027cd09d4b0 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=74918) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 759.072251] env[74918]: DEBUG oslo_concurrency.lockutils [None req-63544f80-a4cf-448b-88ca-5027cd09d4b0 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74918) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 759.072375] env[74918]: DEBUG oslo_concurrency.lockutils [None req-63544f80-a4cf-448b-88ca-5027cd09d4b0 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74918) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 759.072649] env[74918]: DEBUG oslo_concurrency.lockutils [None req-63544f80-a4cf-448b-88ca-5027cd09d4b0 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74918) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 759.089800] env[74918]: DEBUG oslo_concurrency.lockutils [None req-63544f80-a4cf-448b-88ca-5027cd09d4b0 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=74918) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 759.090058] env[74918]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63544f80-a4cf-448b-88ca-5027cd09d4b0 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 759.090196] env[74918]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-63544f80-a4cf-448b-88ca-5027cd09d4b0 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=74918) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 761.094455] env[74918]: CRITICAL neutron [None req-63544f80-a4cf-448b-88ca-5027cd09d4b0 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 761.094455] env[74918]: ERROR neutron Traceback (most recent call last): [ 761.094455] env[74918]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 761.094455] env[74918]: ERROR neutron sys.exit(main()) [ 761.094455] env[74918]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 761.094455] env[74918]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 761.094455] env[74918]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 761.094455] env[74918]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 761.094455] env[74918]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 761.094455] env[74918]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 761.094455] env[74918]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 761.094455] env[74918]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 761.094455] env[74918]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 761.094455] env[74918]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 761.094455] env[74918]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 761.094455] env[74918]: ERROR neutron response = self.get(path=path, params=params) [ 761.095276] env[74918]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 761.095276] env[74918]: ERROR neutron raise RuntimeError(msg, last_err) [ 761.095276] env[74918]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 761.095276] env[74918]: ERROR neutron [ 764.165971] env[75623]: INFO neutron.common.config [-] Logging enabled! [ 764.166292] env[75623]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 764.166523] env[75623]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=75623) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 764.166928] env[75623]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 764.166928] env[75623]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 764.167989] env[75623]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-96c3d0ef-cf91-4647-b025-9ce902fb9f49 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 764.183986] env[75623]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-96c3d0ef-cf91-4647-b025-9ce902fb9f49 None None] Session token - acquiring [ 764.184869] env[75623]: DEBUG networking_nsxv3.common.locking [None req-96c3d0ef-cf91-4647-b025-9ce902fb9f49 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75623) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 764.185117] env[75623]: DEBUG oslo_concurrency.lockutils [None req-96c3d0ef-cf91-4647-b025-9ce902fb9f49 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75623) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 764.185309] env[75623]: DEBUG oslo_concurrency.lockutils [None req-96c3d0ef-cf91-4647-b025-9ce902fb9f49 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75623) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 764.185713] env[75623]: DEBUG oslo_concurrency.lockutils [None req-96c3d0ef-cf91-4647-b025-9ce902fb9f49 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75623) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 764.197511] env[75623]: DEBUG oslo_concurrency.lockutils [None req-96c3d0ef-cf91-4647-b025-9ce902fb9f49 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75623) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 764.197511] env[75623]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-96c3d0ef-cf91-4647-b025-9ce902fb9f49 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 764.197791] env[75623]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-96c3d0ef-cf91-4647-b025-9ce902fb9f49 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=75623) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 766.203957] env[75623]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-96c3d0ef-cf91-4647-b025-9ce902fb9f49 None None] Session token - acquiring [ 766.204281] env[75623]: DEBUG networking_nsxv3.common.locking [None req-96c3d0ef-cf91-4647-b025-9ce902fb9f49 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=75623) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 766.204474] env[75623]: DEBUG oslo_concurrency.lockutils [None req-96c3d0ef-cf91-4647-b025-9ce902fb9f49 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75623) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 766.204598] env[75623]: DEBUG oslo_concurrency.lockutils [None req-96c3d0ef-cf91-4647-b025-9ce902fb9f49 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75623) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 766.204868] env[75623]: DEBUG oslo_concurrency.lockutils [None req-96c3d0ef-cf91-4647-b025-9ce902fb9f49 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75623) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 766.214559] env[75623]: DEBUG oslo_concurrency.lockutils [None req-96c3d0ef-cf91-4647-b025-9ce902fb9f49 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=75623) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 766.214804] env[75623]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-96c3d0ef-cf91-4647-b025-9ce902fb9f49 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 766.214940] env[75623]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-96c3d0ef-cf91-4647-b025-9ce902fb9f49 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=75623) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 768.217068] env[75623]: CRITICAL neutron [None req-96c3d0ef-cf91-4647-b025-9ce902fb9f49 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 768.217068] env[75623]: ERROR neutron Traceback (most recent call last): [ 768.217068] env[75623]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 768.217068] env[75623]: ERROR neutron sys.exit(main()) [ 768.217068] env[75623]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 768.217068] env[75623]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 768.217068] env[75623]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 768.217068] env[75623]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 768.217068] env[75623]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 768.217068] env[75623]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 768.217068] env[75623]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 768.217068] env[75623]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 768.217068] env[75623]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 768.217068] env[75623]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 768.217068] env[75623]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 768.217068] env[75623]: ERROR neutron response = self.get(path=path, params=params) [ 768.217714] env[75623]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 768.217714] env[75623]: ERROR neutron raise RuntimeError(msg, last_err) [ 768.217714] env[75623]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 768.217714] env[75623]: ERROR neutron [ 771.032089] env[76121]: INFO neutron.common.config [-] Logging enabled! [ 771.032449] env[76121]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 771.032603] env[76121]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76121) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 771.032843] env[76121]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 771.032947] env[76121]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 771.033867] env[76121]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-ad0c1704-88a5-42a6-b636-d64f1b6c4c94 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 771.047457] env[76121]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ad0c1704-88a5-42a6-b636-d64f1b6c4c94 None None] Session token - acquiring [ 771.048281] env[76121]: DEBUG networking_nsxv3.common.locking [None req-ad0c1704-88a5-42a6-b636-d64f1b6c4c94 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76121) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 771.048510] env[76121]: DEBUG oslo_concurrency.lockutils [None req-ad0c1704-88a5-42a6-b636-d64f1b6c4c94 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76121) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 771.048640] env[76121]: DEBUG oslo_concurrency.lockutils [None req-ad0c1704-88a5-42a6-b636-d64f1b6c4c94 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76121) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 771.049013] env[76121]: DEBUG oslo_concurrency.lockutils [None req-ad0c1704-88a5-42a6-b636-d64f1b6c4c94 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76121) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 771.058261] env[76121]: DEBUG oslo_concurrency.lockutils [None req-ad0c1704-88a5-42a6-b636-d64f1b6c4c94 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76121) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 771.058503] env[76121]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ad0c1704-88a5-42a6-b636-d64f1b6c4c94 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 771.058656] env[76121]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ad0c1704-88a5-42a6-b636-d64f1b6c4c94 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76121) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 773.065890] env[76121]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ad0c1704-88a5-42a6-b636-d64f1b6c4c94 None None] Session token - acquiring [ 773.066236] env[76121]: DEBUG networking_nsxv3.common.locking [None req-ad0c1704-88a5-42a6-b636-d64f1b6c4c94 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76121) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 773.066443] env[76121]: DEBUG oslo_concurrency.lockutils [None req-ad0c1704-88a5-42a6-b636-d64f1b6c4c94 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76121) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 773.066565] env[76121]: DEBUG oslo_concurrency.lockutils [None req-ad0c1704-88a5-42a6-b636-d64f1b6c4c94 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76121) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 773.066873] env[76121]: DEBUG oslo_concurrency.lockutils [None req-ad0c1704-88a5-42a6-b636-d64f1b6c4c94 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76121) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 773.077919] env[76121]: DEBUG oslo_concurrency.lockutils [None req-ad0c1704-88a5-42a6-b636-d64f1b6c4c94 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76121) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 773.077919] env[76121]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ad0c1704-88a5-42a6-b636-d64f1b6c4c94 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 773.077919] env[76121]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ad0c1704-88a5-42a6-b636-d64f1b6c4c94 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76121) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 775.080322] env[76121]: CRITICAL neutron [None req-ad0c1704-88a5-42a6-b636-d64f1b6c4c94 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 775.080322] env[76121]: ERROR neutron Traceback (most recent call last): [ 775.080322] env[76121]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 775.080322] env[76121]: ERROR neutron sys.exit(main()) [ 775.080322] env[76121]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 775.080322] env[76121]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 775.080322] env[76121]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 775.080322] env[76121]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 775.080322] env[76121]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 775.080322] env[76121]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 775.080322] env[76121]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 775.080322] env[76121]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 775.080322] env[76121]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 775.080322] env[76121]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 775.080322] env[76121]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 775.080322] env[76121]: ERROR neutron response = self.get(path=path, params=params) [ 775.081080] env[76121]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 775.081080] env[76121]: ERROR neutron raise RuntimeError(msg, last_err) [ 775.081080] env[76121]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 775.081080] env[76121]: ERROR neutron [ 777.938016] env[76660]: INFO neutron.common.config [-] Logging enabled! [ 777.938331] env[76660]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 777.938519] env[76660]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76660) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 777.938746] env[76660]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 777.938838] env[76660]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 777.939729] env[76660]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-8dfc4120-e790-435b-b85c-b44cc1d69e61 None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 777.953300] env[76660]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8dfc4120-e790-435b-b85c-b44cc1d69e61 None None] Session token - acquiring [ 777.954042] env[76660]: DEBUG networking_nsxv3.common.locking [None req-8dfc4120-e790-435b-b85c-b44cc1d69e61 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76660) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 777.954239] env[76660]: DEBUG oslo_concurrency.lockutils [None req-8dfc4120-e790-435b-b85c-b44cc1d69e61 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76660) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 777.954365] env[76660]: DEBUG oslo_concurrency.lockutils [None req-8dfc4120-e790-435b-b85c-b44cc1d69e61 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76660) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 777.954719] env[76660]: DEBUG oslo_concurrency.lockutils [None req-8dfc4120-e790-435b-b85c-b44cc1d69e61 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76660) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 777.963909] env[76660]: DEBUG oslo_concurrency.lockutils [None req-8dfc4120-e790-435b-b85c-b44cc1d69e61 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76660) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 777.964163] env[76660]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8dfc4120-e790-435b-b85c-b44cc1d69e61 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 777.964329] env[76660]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8dfc4120-e790-435b-b85c-b44cc1d69e61 None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76660) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 779.971944] env[76660]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8dfc4120-e790-435b-b85c-b44cc1d69e61 None None] Session token - acquiring [ 779.972266] env[76660]: DEBUG networking_nsxv3.common.locking [None req-8dfc4120-e790-435b-b85c-b44cc1d69e61 None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76660) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 779.972455] env[76660]: DEBUG oslo_concurrency.lockutils [None req-8dfc4120-e790-435b-b85c-b44cc1d69e61 None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76660) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 779.972611] env[76660]: DEBUG oslo_concurrency.lockutils [None req-8dfc4120-e790-435b-b85c-b44cc1d69e61 None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76660) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 779.972864] env[76660]: DEBUG oslo_concurrency.lockutils [None req-8dfc4120-e790-435b-b85c-b44cc1d69e61 None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76660) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 779.982556] env[76660]: DEBUG oslo_concurrency.lockutils [None req-8dfc4120-e790-435b-b85c-b44cc1d69e61 None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76660) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 779.982919] env[76660]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8dfc4120-e790-435b-b85c-b44cc1d69e61 None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 779.983069] env[76660]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8dfc4120-e790-435b-b85c-b44cc1d69e61 None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76660) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 781.984860] env[76660]: CRITICAL neutron [None req-8dfc4120-e790-435b-b85c-b44cc1d69e61 None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 781.984860] env[76660]: ERROR neutron Traceback (most recent call last): [ 781.984860] env[76660]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 781.984860] env[76660]: ERROR neutron sys.exit(main()) [ 781.984860] env[76660]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 781.984860] env[76660]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 781.984860] env[76660]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 781.984860] env[76660]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 781.984860] env[76660]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 781.984860] env[76660]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 781.984860] env[76660]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 781.984860] env[76660]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 781.984860] env[76660]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 781.984860] env[76660]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 781.984860] env[76660]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 781.984860] env[76660]: ERROR neutron response = self.get(path=path, params=params) [ 781.985595] env[76660]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 781.985595] env[76660]: ERROR neutron raise RuntimeError(msg, last_err) [ 781.985595] env[76660]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 781.985595] env[76660]: ERROR neutron [ 784.680892] env[76819]: INFO neutron.common.config [-] Logging enabled! [ 784.681188] env[76819]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 784.681462] env[76819]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76819) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 784.681729] env[76819]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 784.681827] env[76819]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 784.682759] env[76819]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-d01d6a1f-c198-47ed-b065-6f0405c16aab None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 784.696556] env[76819]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d01d6a1f-c198-47ed-b065-6f0405c16aab None None] Session token - acquiring [ 784.697333] env[76819]: DEBUG networking_nsxv3.common.locking [None req-d01d6a1f-c198-47ed-b065-6f0405c16aab None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76819) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 784.697556] env[76819]: DEBUG oslo_concurrency.lockutils [None req-d01d6a1f-c198-47ed-b065-6f0405c16aab None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76819) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 784.697689] env[76819]: DEBUG oslo_concurrency.lockutils [None req-d01d6a1f-c198-47ed-b065-6f0405c16aab None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76819) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 784.698053] env[76819]: DEBUG oslo_concurrency.lockutils [None req-d01d6a1f-c198-47ed-b065-6f0405c16aab None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76819) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 784.707793] env[76819]: DEBUG oslo_concurrency.lockutils [None req-d01d6a1f-c198-47ed-b065-6f0405c16aab None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76819) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 784.708093] env[76819]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d01d6a1f-c198-47ed-b065-6f0405c16aab None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 784.708259] env[76819]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d01d6a1f-c198-47ed-b065-6f0405c16aab None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76819) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 786.715763] env[76819]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d01d6a1f-c198-47ed-b065-6f0405c16aab None None] Session token - acquiring [ 786.716110] env[76819]: DEBUG networking_nsxv3.common.locking [None req-d01d6a1f-c198-47ed-b065-6f0405c16aab None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76819) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 786.716306] env[76819]: DEBUG oslo_concurrency.lockutils [None req-d01d6a1f-c198-47ed-b065-6f0405c16aab None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76819) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 786.716432] env[76819]: DEBUG oslo_concurrency.lockutils [None req-d01d6a1f-c198-47ed-b065-6f0405c16aab None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76819) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 786.716709] env[76819]: DEBUG oslo_concurrency.lockutils [None req-d01d6a1f-c198-47ed-b065-6f0405c16aab None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76819) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 786.726419] env[76819]: DEBUG oslo_concurrency.lockutils [None req-d01d6a1f-c198-47ed-b065-6f0405c16aab None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76819) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 786.726692] env[76819]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d01d6a1f-c198-47ed-b065-6f0405c16aab None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 786.726874] env[76819]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-d01d6a1f-c198-47ed-b065-6f0405c16aab None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76819) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 788.728977] env[76819]: CRITICAL neutron [None req-d01d6a1f-c198-47ed-b065-6f0405c16aab None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 788.728977] env[76819]: ERROR neutron Traceback (most recent call last): [ 788.728977] env[76819]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 788.728977] env[76819]: ERROR neutron sys.exit(main()) [ 788.728977] env[76819]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 788.728977] env[76819]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 788.728977] env[76819]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 788.728977] env[76819]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 788.728977] env[76819]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 788.728977] env[76819]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 788.728977] env[76819]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 788.728977] env[76819]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 788.728977] env[76819]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 788.728977] env[76819]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 788.728977] env[76819]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 788.728977] env[76819]: ERROR neutron response = self.get(path=path, params=params) [ 788.729762] env[76819]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 788.729762] env[76819]: ERROR neutron raise RuntimeError(msg, last_err) [ 788.729762] env[76819]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 788.729762] env[76819]: ERROR neutron [ 791.233192] env[76953]: INFO neutron.common.config [-] Logging enabled! [ 791.233591] env[76953]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 791.233676] env[76953]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=76953) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 791.233906] env[76953]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 791.234002] env[76953]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 791.234845] env[76953]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-8276b842-987b-45ff-9408-25e62a461b2e None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 791.248276] env[76953]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8276b842-987b-45ff-9408-25e62a461b2e None None] Session token - acquiring [ 791.248992] env[76953]: DEBUG networking_nsxv3.common.locking [None req-8276b842-987b-45ff-9408-25e62a461b2e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76953) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 791.249192] env[76953]: DEBUG oslo_concurrency.lockutils [None req-8276b842-987b-45ff-9408-25e62a461b2e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76953) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 791.249320] env[76953]: DEBUG oslo_concurrency.lockutils [None req-8276b842-987b-45ff-9408-25e62a461b2e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76953) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 791.249700] env[76953]: DEBUG oslo_concurrency.lockutils [None req-8276b842-987b-45ff-9408-25e62a461b2e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76953) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 791.258777] env[76953]: DEBUG oslo_concurrency.lockutils [None req-8276b842-987b-45ff-9408-25e62a461b2e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76953) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 791.259115] env[76953]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8276b842-987b-45ff-9408-25e62a461b2e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 791.259235] env[76953]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8276b842-987b-45ff-9408-25e62a461b2e None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=76953) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 793.267169] env[76953]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8276b842-987b-45ff-9408-25e62a461b2e None None] Session token - acquiring [ 793.267540] env[76953]: DEBUG networking_nsxv3.common.locking [None req-8276b842-987b-45ff-9408-25e62a461b2e None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=76953) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 793.267736] env[76953]: DEBUG oslo_concurrency.lockutils [None req-8276b842-987b-45ff-9408-25e62a461b2e None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76953) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 793.267789] env[76953]: DEBUG oslo_concurrency.lockutils [None req-8276b842-987b-45ff-9408-25e62a461b2e None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76953) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 793.268071] env[76953]: DEBUG oslo_concurrency.lockutils [None req-8276b842-987b-45ff-9408-25e62a461b2e None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76953) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 793.277655] env[76953]: DEBUG oslo_concurrency.lockutils [None req-8276b842-987b-45ff-9408-25e62a461b2e None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=76953) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 793.277917] env[76953]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8276b842-987b-45ff-9408-25e62a461b2e None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 793.278071] env[76953]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-8276b842-987b-45ff-9408-25e62a461b2e None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=76953) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 795.280420] env[76953]: CRITICAL neutron [None req-8276b842-987b-45ff-9408-25e62a461b2e None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 795.280420] env[76953]: ERROR neutron Traceback (most recent call last): [ 795.280420] env[76953]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 795.280420] env[76953]: ERROR neutron sys.exit(main()) [ 795.280420] env[76953]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 795.280420] env[76953]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 795.280420] env[76953]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 795.280420] env[76953]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 795.280420] env[76953]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 795.280420] env[76953]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 795.280420] env[76953]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 795.280420] env[76953]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 795.280420] env[76953]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 795.280420] env[76953]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 795.280420] env[76953]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 795.280420] env[76953]: ERROR neutron response = self.get(path=path, params=params) [ 795.281324] env[76953]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 795.281324] env[76953]: ERROR neutron raise RuntimeError(msg, last_err) [ 795.281324] env[76953]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 795.281324] env[76953]: ERROR neutron [ 797.731852] env[77019]: INFO neutron.common.config [-] Logging enabled! [ 797.732268] env[77019]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 797.732312] env[77019]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77019) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 797.732517] env[77019]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 797.732627] env[77019]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 797.733460] env[77019]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-75d0a11f-9287-49a1-9a24-1ec395b6758c None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 797.747250] env[77019]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-75d0a11f-9287-49a1-9a24-1ec395b6758c None None] Session token - acquiring [ 797.747992] env[77019]: DEBUG networking_nsxv3.common.locking [None req-75d0a11f-9287-49a1-9a24-1ec395b6758c None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77019) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 797.748198] env[77019]: DEBUG oslo_concurrency.lockutils [None req-75d0a11f-9287-49a1-9a24-1ec395b6758c None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77019) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 797.748330] env[77019]: DEBUG oslo_concurrency.lockutils [None req-75d0a11f-9287-49a1-9a24-1ec395b6758c None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77019) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 797.748674] env[77019]: DEBUG oslo_concurrency.lockutils [None req-75d0a11f-9287-49a1-9a24-1ec395b6758c None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77019) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 797.758093] env[77019]: DEBUG oslo_concurrency.lockutils [None req-75d0a11f-9287-49a1-9a24-1ec395b6758c None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77019) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 797.758385] env[77019]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-75d0a11f-9287-49a1-9a24-1ec395b6758c None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 797.758594] env[77019]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-75d0a11f-9287-49a1-9a24-1ec395b6758c None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77019) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 799.766573] env[77019]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-75d0a11f-9287-49a1-9a24-1ec395b6758c None None] Session token - acquiring [ 799.766948] env[77019]: DEBUG networking_nsxv3.common.locking [None req-75d0a11f-9287-49a1-9a24-1ec395b6758c None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77019) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 799.767020] env[77019]: DEBUG oslo_concurrency.lockutils [None req-75d0a11f-9287-49a1-9a24-1ec395b6758c None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77019) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 799.767160] env[77019]: DEBUG oslo_concurrency.lockutils [None req-75d0a11f-9287-49a1-9a24-1ec395b6758c None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77019) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 799.767429] env[77019]: DEBUG oslo_concurrency.lockutils [None req-75d0a11f-9287-49a1-9a24-1ec395b6758c None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77019) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 799.777180] env[77019]: DEBUG oslo_concurrency.lockutils [None req-75d0a11f-9287-49a1-9a24-1ec395b6758c None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77019) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 799.777560] env[77019]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-75d0a11f-9287-49a1-9a24-1ec395b6758c None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 799.777734] env[77019]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-75d0a11f-9287-49a1-9a24-1ec395b6758c None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77019) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 801.780808] env[77019]: CRITICAL neutron [None req-75d0a11f-9287-49a1-9a24-1ec395b6758c None None] Unhandled error: RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 801.780808] env[77019]: ERROR neutron Traceback (most recent call last): [ 801.780808] env[77019]: ERROR neutron File "/usr/local/bin/neutron-nsxv3-agent", line 10, in [ 801.780808] env[77019]: ERROR neutron sys.exit(main()) [ 801.780808] env[77019]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 331, in main [ 801.780808] env[77019]: ERROR neutron NSXv3Manager(rpc=nsxv3_rpc.NSXv3ServerRpcApi()), [ 801.780808] env[77019]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/agent.py", line 145, in __init__ [ 801.780808] env[77019]: ERROR neutron self.mngr_provider = provider_nsx_mgmt.Provider() [ 801.780808] env[77019]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 443, in __init__ [ 801.780808] env[77019]: ERROR neutron super(Provider, self).__init__(client=Client(), zone_id=None) [ 801.780808] env[77019]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider.py", line 207, in __init__ [ 801.780808] env[77019]: ERROR neutron self.zone_id, self.zone_tags = self._load_zones() [ 801.780808] env[77019]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/provider_nsx_mgmt.py", line 454, in _load_zones [ 801.780808] env[77019]: ERROR neutron for zone in self.client.get_all(path="/api/v1/transport-zones"): [ 801.780808] env[77019]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 276, in get_all [ 801.780808] env[77019]: ERROR neutron response = self.get(path=path, params=params) [ 801.781979] env[77019]: ERROR neutron File "/opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py", line 129, in decorator [ 801.781979] env[77019]: ERROR neutron raise RuntimeError(msg, last_err) [ 801.781979] env[77019]: ERROR neutron RuntimeError: ('Retrying connection (2/2) with timeout 2s for Client.get', HTTPError('403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create')) [ 801.781979] env[77019]: ERROR neutron [ 804.258787] env[77026]: INFO neutron.common.config [-] Logging enabled! [ 804.259232] env[77026]: INFO neutron.common.config [-] /usr/local/bin/neutron-nsxv3-agent version 0.0.1.dev57 [ 804.259285] env[77026]: DEBUG neutron.common.config [-] command line: /usr/local/bin/neutron-nsxv3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2-nsxv3-1.ini {{(pid=77026) setup_logging /opt/stack/neutron/neutron/common/config.py:123}} [ 804.259512] env[77026]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] VMware NSXv3 Agent initializing ... [ 804.259612] env[77026]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.agent [-] Eventlet blocking behavior detection initialization completed. [ 804.260500] env[77026]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.provider_nsx_mgmt [None req-ca6c64f0-ece7-470a-980e-088978b2b10d None None] Looking for TransportZone with name nsx-vlan-transportzone. [ 804.273871] env[77026]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ca6c64f0-ece7-470a-980e-088978b2b10d None None] Session token - acquiring [ 804.274599] env[77026]: DEBUG networking_nsxv3.common.locking [None req-ca6c64f0-ece7-470a-980e-088978b2b10d None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77026) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 804.274802] env[77026]: DEBUG oslo_concurrency.lockutils [None req-ca6c64f0-ece7-470a-980e-088978b2b10d None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77026) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 804.274931] env[77026]: DEBUG oslo_concurrency.lockutils [None req-ca6c64f0-ece7-470a-980e-088978b2b10d None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77026) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 804.275283] env[77026]: DEBUG oslo_concurrency.lockutils [None req-ca6c64f0-ece7-470a-980e-088978b2b10d None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77026) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 804.284267] env[77026]: DEBUG oslo_concurrency.lockutils [None req-ca6c64f0-ece7-470a-980e-088978b2b10d None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77026) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 804.284526] env[77026]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ca6c64f0-ece7-470a-980e-088978b2b10d None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 804.284676] env[77026]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ca6c64f0-ece7-470a-980e-088978b2b10d None None] Retrying connection (1/2) with timeout 2s for Client.get {{(pid=77026) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}} [ 806.290906] env[77026]: INFO networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ca6c64f0-ece7-470a-980e-088978b2b10d None None] Session token - acquiring [ 806.291209] env[77026]: DEBUG networking_nsxv3.common.locking [None req-ca6c64f0-ece7-470a-980e-088978b2b10d None None] Lock https://nsxt.osci.c.eu-de-1.cloud.sap:443 taken with stack trace [>, , , , , , , , , ] {{(pid=77026) get_lock /opt/stack/networking-nsx-t/networking_nsxv3/common/locking.py:46}} [ 806.291380] env[77026]: DEBUG oslo_concurrency.lockutils [None req-ca6c64f0-ece7-470a-980e-088978b2b10d None None] Acquiring lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77026) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:312}} [ 806.291506] env[77026]: DEBUG oslo_concurrency.lockutils [None req-ca6c64f0-ece7-470a-980e-088978b2b10d None None] Acquired lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77026) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:315}} [ 806.291797] env[77026]: DEBUG oslo_concurrency.lockutils [None req-ca6c64f0-ece7-470a-980e-088978b2b10d None None] Acquired external semaphore "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77026) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:323}} [ 806.300933] env[77026]: DEBUG oslo_concurrency.lockutils [None req-ca6c64f0-ece7-470a-980e-088978b2b10d None None] Releasing lock "https://nsxt.osci.c.eu-de-1.cloud.sap:443" {{(pid=77026) lock /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:333}} [ 806.301184] env[77026]: ERROR networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ca6c64f0-ece7-470a-980e-088978b2b10d None None] Request=Function GET Argumetns {'path': '/api/v1/transport-zones', 'params': {'page_size': 200, 'cursor': ''}} Response=403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nsxt.osci.c.eu-de-1.cloud.sap:443/api/session/create [ 806.301307] env[77026]: DEBUG networking_nsxv3.plugins.ml2.drivers.nsxv3.agent.client_nsx [None req-ca6c64f0-ece7-470a-980e-088978b2b10d None None] Retrying connection (2/2) with timeout 2s for Client.get {{(pid=77026) decorator /opt/stack/networking-nsx-t/networking_nsxv3/plugins/ml2/drivers/nsxv3/agent/client_nsx.py:126}}