[ 392.581443] nova-conductor[54261]: INFO dbcounter [None req-79af25d7-0948-46e9-830e-047783347c91 None None] Registered counter for database nova_api [ 392.590236] nova-conductor[54261]: DEBUG oslo_db.sqlalchemy.engines [None req-79af25d7-0948-46e9-830e-047783347c91 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=54261) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:342}} [ 392.618226] nova-conductor[54261]: DEBUG dbcounter [-] [54261] Writer thread running {{(pid=54261) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} [ 392.620212] nova-conductor[54261]: DEBUG nova.context [None req-79af25d7-0948-46e9-830e-047783347c91 None None] Found 2 cells: 00000000-0000-0000-0000-000000000000(cell0),2d4e634d-6292-45c4-9eb1-09b106cacc48(cell1) {{(pid=54261) load_cells /opt/stack/nova/nova/context.py:464}} [ 392.621805] nova-conductor[54261]: DEBUG oslo_concurrency.lockutils [None req-79af25d7-0948-46e9-830e-047783347c91 None None] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54261) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 392.622003] nova-conductor[54261]: DEBUG oslo_concurrency.lockutils [None req-79af25d7-0948-46e9-830e-047783347c91 None None] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54261) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 392.622431] nova-conductor[54261]: DEBUG oslo_concurrency.lockutils [None req-79af25d7-0948-46e9-830e-047783347c91 None None] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54261) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 392.622821] nova-conductor[54261]: DEBUG oslo_concurrency.lockutils [None req-79af25d7-0948-46e9-830e-047783347c91 None None] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54261) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 392.623003] nova-conductor[54261]: DEBUG oslo_concurrency.lockutils [None req-79af25d7-0948-46e9-830e-047783347c91 None None] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54261) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 392.623936] nova-conductor[54261]: DEBUG oslo_concurrency.lockutils [None req-79af25d7-0948-46e9-830e-047783347c91 None None] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=54261) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 392.625196] nova-conductor[54261]: INFO dbcounter [None req-79af25d7-0948-46e9-830e-047783347c91 None None] Registered counter for database nova_cell0 [ 392.627450] nova-conductor[54261]: INFO dbcounter [None req-79af25d7-0948-46e9-830e-047783347c91 None None] Registered counter for database nova_cell1 [ 392.630142] nova-conductor[54261]: DEBUG oslo_db.sqlalchemy.engines [None req-79af25d7-0948-46e9-830e-047783347c91 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=54261) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:342}} [ 392.630472] nova-conductor[54261]: DEBUG oslo_db.sqlalchemy.engines [None req-79af25d7-0948-46e9-830e-047783347c91 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=54261) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:342}} [ 392.705282] nova-conductor[54261]: DEBUG dbcounter [-] [54261] Writer thread running {{(pid=54261) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} [ 392.708353] nova-conductor[54261]: DEBUG dbcounter [-] [54261] Writer thread running {{(pid=54261) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} [ 392.710011] nova-conductor[54261]: DEBUG oslo_concurrency.lockutils [None req-79af25d7-0948-46e9-830e-047783347c91 None None] Acquiring lock "singleton_lock" {{(pid=54261) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 392.710175] nova-conductor[54261]: DEBUG oslo_concurrency.lockutils [None req-79af25d7-0948-46e9-830e-047783347c91 None None] Acquired lock "singleton_lock" {{(pid=54261) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 392.710405] nova-conductor[54261]: DEBUG oslo_concurrency.lockutils [None req-79af25d7-0948-46e9-830e-047783347c91 None None] Releasing lock "singleton_lock" {{(pid=54261) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 392.710790] nova-conductor[54261]: INFO oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] Starting 2 workers [ 392.714925] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] Started child 54691 {{(pid=54261) _start_child /opt/stack/data/venv/lib/python3.10/site-packages/oslo_service/service.py:579}} [ 392.718505] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] Started child 54692 {{(pid=54261) _start_child /opt/stack/data/venv/lib/python3.10/site-packages/oslo_service/service.py:579}} [ 392.718879] nova-conductor[54691]: INFO nova.service [-] Starting conductor node (version 0.0.1) [ 392.719137] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] Full set of CONF: {{(pid=54261) wait /opt/stack/data/venv/lib/python3.10/site-packages/oslo_service/service.py:653}} [ 392.719326] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ******************************************************************************** {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2600}} [ 392.719474] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] Configuration options gathered from: {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2601}} [ 392.719635] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] command line args: ['--config-file', '/etc/nova/nova.conf'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2602}} [ 392.720114] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] config files: ['/etc/nova/nova.conf'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2603}} [ 392.720259] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ================================================================================ {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2605}} [ 392.720644] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] allow_resize_to_same_host = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.720844] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] arq_binding_timeout = 300 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.721038] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] block_device_allocate_retries = 60 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.721218] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] block_device_allocate_retries_interval = 3 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.721551] nova-conductor[54692]: INFO nova.service [-] Starting conductor node (version 0.0.1) [ 392.721748] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cert = self.pem {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.721936] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] compute_driver = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.722180] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] compute_monitors = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.722374] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] config_dir = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.722577] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] config_drive_format = iso9660 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.722711] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] config_file = ['/etc/nova/nova.conf'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.722900] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] config_source = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.723083] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] console_host = devstack {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.723268] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] control_exchange = nova {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.723463] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cpu_allocation_ratio = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.723622] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] daemon = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.723812] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] debug = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.723980] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] default_access_ip_network_name = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.724178] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] default_availability_zone = nova {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.724345] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] default_ephemeral_format = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.724526] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] default_green_pool_size = 1000 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.724797] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'glanceclient=WARN', 'oslo.privsep.daemon=INFO'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.724986] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] default_schedule_zone = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.725146] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] disk_allocation_ratio = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.725443] nova-conductor[54691]: INFO dbcounter [None req-5a03865e-6bbe-4415-b1e4-2c0f3afaee55 None None] Registered counter for database nova_cell1 [ 392.725586] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] enable_new_services = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.725825] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] enabled_apis = ['osapi_compute'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.726014] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] enabled_ssl_apis = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.726204] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] flat_injected = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.726366] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] force_config_drive = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.726538] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] force_raw_images = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.726718] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] graceful_shutdown_timeout = 5 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.726876] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] heal_instance_info_cache_interval = 60 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.727139] nova-conductor[54692]: INFO dbcounter [None req-9741ab19-8a44-45b2-bebf-12c0547d4c1a None None] Registered counter for database nova_cell1 [ 392.727343] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] host = devstack {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.727552] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] initial_cpu_allocation_ratio = 4.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.727728] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] initial_disk_allocation_ratio = 1.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.727913] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] initial_ram_allocation_ratio = 1.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.728178] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] injected_network_template = /opt/stack/nova/nova/virt/interfaces.template {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.728348] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] instance_build_timeout = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.728502] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] instance_delete_interval = 300 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.728661] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] instance_format = [instance: %(uuid)s] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.728846] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] instance_name_template = instance-%08x {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.729001] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] instance_usage_audit = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.729265] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] instance_usage_audit_period = month {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.729426] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.729602] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] instances_path = /opt/stack/data/nova/instances {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.729760] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] internal_service_availability_zone = internal {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.729927] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] key = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.730102] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] live_migration_retry_count = 30 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.730279] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] log_color = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.730453] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] log_config_append = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.730616] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.730822] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] log_dir = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.730997] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] log_file = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.731119] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] log_options = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.731313] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] log_rotate_interval = 1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.731504] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] log_rotate_interval_type = days {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.731682] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] log_rotation_type = none {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.731907] nova-conductor[54691]: DEBUG oslo_db.sqlalchemy.engines [None req-5a03865e-6bbe-4415-b1e4-2c0f3afaee55 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=54691) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:342}} [ 392.732063] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.732205] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.732382] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.732603] nova-conductor[54692]: DEBUG oslo_db.sqlalchemy.engines [None req-9741ab19-8a44-45b2-bebf-12c0547d4c1a None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=54692) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:342}} [ 392.732738] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.732874] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.733437] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] long_rpc_timeout = 1800 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.733597] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] max_concurrent_builds = 10 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.733771] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] max_concurrent_live_migrations = 1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.733925] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] max_concurrent_snapshots = 5 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.734083] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] max_local_block_devices = 3 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.734235] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] max_logfile_count = 30 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.734384] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] max_logfile_size_mb = 200 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.734560] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] maximum_instance_delete_attempts = 5 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.734732] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] metadata_listen = 0.0.0.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.734925] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] metadata_listen_port = 8775 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.735104] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] metadata_workers = 2 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.735259] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] migrate_max_retries = -1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.735414] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] mkisofs_cmd = genisoimage {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.735634] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] my_block_storage_ip = 10.180.1.21 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.735759] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] my_ip = 10.180.1.21 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.735917] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] network_allocate_retries = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.736104] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] non_inheritable_image_properties = ['cache_in_nova', 'bittorrent'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.736279] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] osapi_compute_listen = 0.0.0.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.736437] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] osapi_compute_listen_port = 8774 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.736595] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] osapi_compute_unique_server_name_scope = {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.736758] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] osapi_compute_workers = 2 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.736917] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] password_length = 12 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.737072] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] periodic_enable = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.737233] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] periodic_fuzzy_delay = 60 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.737399] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] pointer_model = usbtablet {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.737585] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] preallocate_images = none {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.737740] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] publish_errors = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.737863] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] pybasedir = /opt/stack/nova {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.738017] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ram_allocation_ratio = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.738172] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] rate_limit_burst = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.738328] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] rate_limit_except_level = CRITICAL {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.738482] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] rate_limit_interval = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.738637] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] reboot_timeout = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.738805] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] reclaim_instance_interval = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.738953] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] record = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.739106] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] reimage_timeout_per_gb = 20 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.739279] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] report_interval = 120 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.739434] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] rescue_timeout = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.739587] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] reserved_host_cpus = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.739743] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] reserved_host_disk_mb = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.739894] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] reserved_host_memory_mb = 512 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.740088] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] reserved_huge_pages = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.740250] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] resize_confirm_window = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.740439] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] resize_fs_using_block_device = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.740598] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] resume_guests_state_on_host_boot = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.740774] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] rootwrap_config = /etc/nova/rootwrap.conf {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.740954] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] rpc_response_timeout = 60 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.741110] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] run_external_periodic_tasks = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.741287] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] running_deleted_instance_action = reap {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.741444] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] running_deleted_instance_poll_interval = 1800 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.741594] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] running_deleted_instance_timeout = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.741767] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] scheduler_instance_sync_interval = 120 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.741927] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] service_down_time = 720 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.742114] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] servicegroup_driver = db {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.742276] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] shelved_offload_time = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.742430] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] shelved_poll_interval = 3600 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.742609] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] shutdown_timeout = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.742763] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] source_is_ipv6 = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.742911] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writer thread running {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} [ 392.743053] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ssl_only = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.743233] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] state_path = /opt/stack/data/nova {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.743394] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] sync_power_state_interval = 600 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.743572] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writer thread running {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} [ 392.743705] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] sync_power_state_pool_size = 1000 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.743884] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] syslog_log_facility = LOG_USER {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.744055] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] tempdir = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.744220] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] timeout_nbd = 10 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.744382] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] transport_url = **** {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.744536] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] update_resources_interval = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.744689] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] use_cow_images = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.744840] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] use_eventlog = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.744997] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] use_journal = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.745149] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] use_json = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.745300] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] use_rootwrap_daemon = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.745448] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] use_stderr = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.745595] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] use_syslog = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.745743] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vcpu_pin_set = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.745903] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vif_plugging_is_fatal = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.746061] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vif_plugging_timeout = 300 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.746253] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] virt_mkfs = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.746407] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] volume_usage_poll_interval = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.746556] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] watch_log_file = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.746730] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] web = /usr/share/spice-html5 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 392.746993] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_concurrency.disable_process_locking = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.747188] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_concurrency.lock_path = /opt/stack/data/nova {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.747386] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.747545] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.747708] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_metrics.metrics_process_name = {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.747865] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.748029] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.748247] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api.auth_strategy = keystone {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.748427] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api.compute_link_prefix = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.748629] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api.config_drive_skip_versions = 1.0 2007-01-19 2007-03-01 2007-08-29 2007-10-10 2007-12-15 2008-02-01 2008-09-01 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.748800] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api.dhcp_domain = novalocal {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.748964] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api.enable_instance_password = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.749116] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api.glance_link_prefix = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.749290] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api.instance_list_cells_batch_fixed_size = 100 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.749473] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api.instance_list_cells_batch_strategy = distributed {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.749631] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api.instance_list_per_project_cells = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.749784] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api.list_records_by_skipping_down_cells = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.749942] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api.local_metadata_per_cell = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.750103] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api.max_limit = 1000 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.750283] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api.metadata_cache_expiration = 15 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.750451] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api.neutron_default_tenant_id = default {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.750617] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api.response_validation = warn {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.750777] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api.use_neutron_default_nets = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.750940] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api.vendordata_dynamic_connect_timeout = 5 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.751094] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api.vendordata_dynamic_failure_fatal = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.751250] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api.vendordata_dynamic_read_timeout = 5 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.751412] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api.vendordata_dynamic_ssl_certfile = {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.751579] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api.vendordata_dynamic_targets = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.751739] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api.vendordata_jsonfile_path = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.751911] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api.vendordata_providers = ['StaticJSON'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.752187] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.backend = dogpile.cache.memcached {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.752376] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.backend_argument = **** {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.752561] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.config_prefix = cache.oslo {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.752754] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.dead_timeout = 60.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.752914] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.debug_cache_backend = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.753088] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.enable_retry_client = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.753242] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.enable_socket_keepalive = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.753408] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.enabled = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.753568] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.enforce_fips_mode = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.753746] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.expiration_time = 600 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.753904] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.hashclient_retry_attempts = 2 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.754066] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.hashclient_retry_delay = 1.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.754221] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.memcache_dead_retry = 300 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.754392] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.memcache_password = **** {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.754548] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.memcache_pool_connection_get_timeout = 10 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.754700] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.memcache_pool_flush_on_reconnect = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.754856] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.memcache_pool_maxsize = 10 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.755003] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.memcache_pool_unused_timeout = 60 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.755155] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.memcache_sasl_enabled = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.755334] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.memcache_servers = ['localhost:11211'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.755506] nova-conductor[54691]: DEBUG nova.service [None req-5a03865e-6bbe-4415-b1e4-2c0f3afaee55 None None] Creating RPC server for service conductor {{(pid=54691) start /opt/stack/nova/nova/service.py:186}} [ 392.755646] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.memcache_socket_timeout = 1.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.755827] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.memcache_username = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.755991] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.proxies = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.756168] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.redis_db = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.756322] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.redis_password = **** {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.756484] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.redis_sentinel_service_name = mymaster {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.756649] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.redis_sentinels = ['localhost:26379'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.756810] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.redis_server = localhost:6379 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.756964] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.redis_socket_timeout = 1.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.757119] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.redis_username = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.757276] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.retry_attempts = 2 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.757432] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.retry_delay = 0.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.758015] nova-conductor[54692]: DEBUG nova.service [None req-9741ab19-8a44-45b2-bebf-12c0547d4c1a None None] Creating RPC server for service conductor {{(pid=54692) start /opt/stack/nova/nova/service.py:186}} [ 392.758087] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.socket_keepalive_count = 1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.758087] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.socket_keepalive_idle = 1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.758145] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.socket_keepalive_interval = 1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.758255] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.tls_allowed_ciphers = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.758416] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.tls_cafile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.758570] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.tls_certfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.758730] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.tls_enabled = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.758886] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cache.tls_keyfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.759095] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cinder.auth_section = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.759288] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cinder.auth_type = password {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.759470] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cinder.cafile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.759658] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cinder.catalog_info = volumev3::publicURL {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.759825] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cinder.certfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.759990] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cinder.collect_timing = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.760171] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cinder.cross_az_attach = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.760328] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cinder.debug = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.760501] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cinder.endpoint_template = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.760662] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cinder.http_retries = 3 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.760819] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cinder.insecure = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.760977] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cinder.keyfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.761150] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cinder.os_region_name = RegionOne {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.761310] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cinder.split_loggers = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.761471] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cinder.timeout = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.761645] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] compute.consecutive_build_service_disable_threshold = 10 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.761798] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] compute.cpu_dedicated_set = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.761954] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] compute.cpu_shared_set = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.762136] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] compute.image_type_exclude_list = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.762299] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] compute.live_migration_wait_for_vif_plug = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.762460] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] compute.max_concurrent_disk_ops = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.762612] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] compute.max_disk_devices_to_attach = -1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.762778] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] compute.packing_host_numa_cells_allocation_strategy = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.762945] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] compute.provider_config_location = /etc/nova/provider_config/ {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.763106] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] compute.resource_provider_association_refresh = 300 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.763266] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] compute.sharing_providers_max_uuids_per_request = 200 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.763428] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] compute.shutdown_retry_interval = 10 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.763607] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] compute.vmdk_allowed_types = ['streamOptimized', 'monolithicSparse'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.763804] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] conductor.workers = 2 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.763978] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] console.allowed_origins = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.764150] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] console.ssl_ciphers = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.764325] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] console.ssl_minimum_version = default {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.764490] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] consoleauth.enforce_session_timeout = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.764655] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] consoleauth.token_ttl = 600 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.764849] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cyborg.cafile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.765006] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cyborg.certfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.765166] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cyborg.collect_timing = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.765323] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cyborg.connect_retries = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.765500] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cyborg.connect_retry_delay = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.765654] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cyborg.endpoint_override = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.765809] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cyborg.insecure = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.765958] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cyborg.keyfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.766113] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cyborg.max_version = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.766262] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cyborg.min_version = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.766412] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cyborg.region_name = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.766561] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cyborg.retriable_status_codes = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.766710] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cyborg.service_name = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.766961] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cyborg.service_type = accelerator {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.767141] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cyborg.split_loggers = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.767294] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cyborg.status_code_retries = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.767478] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cyborg.status_code_retry_delay = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.767643] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cyborg.timeout = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.767784] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cyborg.valid_interfaces = ['internal', 'public'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.767925] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] cyborg.version = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.768133] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] database.backend = sqlalchemy {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.768295] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] database.connection = **** {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.768658] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] database.connection_debug = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.768658] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] database.connection_parameters = {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.768848] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] database.connection_recycle_time = 3600 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.769254] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] database.connection_trace = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.769254] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] database.db_inc_retry_interval = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.769470] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] database.db_max_retries = 20 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.769582] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] database.db_max_retry_interval = 10 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.769750] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] database.db_retry_interval = 1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.769897] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] database.max_overflow = 50 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.770062] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] database.max_pool_size = 5 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.770201] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] database.max_retries = 10 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.770577] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] database.mysql_sql_mode = TRADITIONAL {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.770642] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] database.mysql_wsrep_sync_wait = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.770760] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] database.pool_timeout = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.770921] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] database.retry_interval = 10 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.771584] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] database.slave_connection = **** {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.771584] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] database.sqlite_synchronous = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.771584] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] database.use_db_reconnect = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.771584] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api_database.backend = sqlalchemy {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.771750] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api_database.connection = **** {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.771906] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api_database.connection_debug = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.772089] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api_database.connection_parameters = {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.772260] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api_database.connection_recycle_time = 3600 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.772423] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api_database.connection_trace = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.772599] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api_database.db_inc_retry_interval = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.772820] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api_database.db_max_retries = 20 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.773292] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api_database.db_max_retry_interval = 10 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.773292] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api_database.db_retry_interval = 1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.773292] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api_database.max_overflow = 50 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.773498] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api_database.max_pool_size = 5 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.773636] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api_database.max_retries = 10 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.773776] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api_database.mysql_sql_mode = TRADITIONAL {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.773934] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api_database.mysql_wsrep_sync_wait = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.774116] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api_database.pool_timeout = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.774299] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api_database.retry_interval = 10 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.774434] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api_database.slave_connection = **** {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.774671] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] api_database.sqlite_synchronous = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.774774] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] devices.enabled_mdev_types = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.774965] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ephemeral_storage_encryption.cipher = aes-xts-plain64 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.775121] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ephemeral_storage_encryption.default_format = luks {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.775289] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ephemeral_storage_encryption.enabled = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.775463] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ephemeral_storage_encryption.key_size = 512 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.775682] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.api_servers = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.775855] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.cafile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.776010] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.certfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.776190] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.collect_timing = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.776373] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.connect_retries = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.776530] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.connect_retry_delay = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.776710] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.debug = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.777018] nova-conductor[54691]: INFO oslo.messaging._drivers.impl_rabbit [None req-5a03865e-6bbe-4415-b1e4-2c0f3afaee55 None None] Creating fanout queue: conductor_fanout_b6e7e44ca9a14fc186c815ffc10c1be5 [ 392.777162] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.default_trusted_certificate_ids = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.777370] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.enable_certificate_validation = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.777538] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.enable_rbd_download = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.777722] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.endpoint_override = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.777895] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.insecure = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.778051] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.keyfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.778210] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.max_version = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.778364] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.min_version = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.778523] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.num_retries = 3 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.778688] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.rbd_ceph_conf = {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.778851] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.rbd_connect_timeout = 5 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.779015] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.rbd_pool = {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.779187] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.rbd_user = {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.779342] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.region_name = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.779496] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.retriable_status_codes = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.779681] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.service_name = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.779849] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.service_type = image {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.780019] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.split_loggers = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.780185] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.status_code_retries = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.780338] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.status_code_retry_delay = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.780523] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.timeout = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.780757] nova-conductor[54691]: DEBUG nova.service [None req-5a03865e-6bbe-4415-b1e4-2c0f3afaee55 None None] Join ServiceGroup membership for this service conductor {{(pid=54691) start /opt/stack/nova/nova/service.py:203}} [ 392.780900] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.valid_interfaces = ['internal', 'public'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.781131] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.verify_glance_signatures = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.781273] nova-conductor[54691]: DEBUG nova.servicegroup.drivers.db [None req-5a03865e-6bbe-4415-b1e4-2c0f3afaee55 None None] DB_Driver: join new ServiceGroup member devstack to the conductor group, service = {{(pid=54691) join /opt/stack/nova/nova/servicegroup/drivers/db.py:44}} [ 392.781409] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] glance.version = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.781604] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] guestfs.debug = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.781806] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] mks.enabled = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.782372] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] mks.mksproxy_base_url = http://127.0.0.1:6090/ {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.782592] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] image_cache.manager_interval = 2400 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.782779] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] image_cache.precache_concurrency = 1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.782950] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] image_cache.remove_unused_base_images = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.783121] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] image_cache.remove_unused_original_minimum_age_seconds = 86400 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.783289] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] image_cache.remove_unused_resized_minimum_age_seconds = 3600 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.783465] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] image_cache.subdirectory_name = _base {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.783661] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.api_max_retries = 60 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.783876] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.api_retry_interval = 2 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.784067] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.auth_section = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.784190] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.auth_type = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.784341] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.cafile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.784500] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.certfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.784688] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.collect_timing = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.784949] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.conductor_group = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.785413] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.connect_retries = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.785413] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.connect_retry_delay = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.785413] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.endpoint_override = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.785531] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.insecure = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.785691] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.keyfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.785845] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.max_version = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.786167] nova-conductor[54692]: INFO oslo.messaging._drivers.impl_rabbit [None req-9741ab19-8a44-45b2-bebf-12c0547d4c1a None None] Creating fanout queue: conductor_fanout_6e0a3c46c45e4db683be62d07e5842f5 [ 392.786305] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.min_version = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.786516] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.peer_list = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.786928] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.region_name = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.786928] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.retriable_status_codes = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.787025] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.serial_console_state_timeout = 10 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.787187] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.service_name = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.787344] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.service_type = baremetal {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.787507] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.shard = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.787668] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.split_loggers = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.787823] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.status_code_retries = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.787977] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.status_code_retry_delay = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.788150] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.timeout = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.788328] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.valid_interfaces = ['internal', 'public'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.788502] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ironic.version = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.788977] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] key_manager.backend = nova.keymgr.conf_key_mgr.ConfKeyManager {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.788977] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] key_manager.fixed_key = **** {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.789119] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican.auth_endpoint = http://localhost/identity/v3 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.789284] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican.barbican_api_version = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.789507] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican.barbican_endpoint = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.789703] nova-conductor[54692]: DEBUG nova.service [None req-9741ab19-8a44-45b2-bebf-12c0547d4c1a None None] Join ServiceGroup membership for this service conductor {{(pid=54692) start /opt/stack/nova/nova/service.py:203}} [ 392.789842] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican.barbican_endpoint_type = public {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.790057] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican.barbican_region_name = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.790234] nova-conductor[54692]: DEBUG nova.servicegroup.drivers.db [None req-9741ab19-8a44-45b2-bebf-12c0547d4c1a None None] DB_Driver: join new ServiceGroup member devstack to the conductor group, service = {{(pid=54692) join /opt/stack/nova/nova/servicegroup/drivers/db.py:44}} [ 392.790371] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican.cafile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.790544] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican.certfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.790819] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican.collect_timing = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.790908] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican.insecure = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.791057] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican.keyfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.791218] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican.number_of_retries = 60 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.791432] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican.retry_delay = 1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.791542] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican.send_service_user_token = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.791700] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican.split_loggers = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.791854] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican.timeout = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.792029] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican.verify_ssl = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.792196] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican.verify_ssl_path = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.792386] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican_service_user.auth_section = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.792548] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican_service_user.auth_type = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.792705] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican_service_user.cafile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.792857] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican_service_user.certfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.793016] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican_service_user.collect_timing = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.793176] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican_service_user.insecure = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.793330] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican_service_user.keyfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.793490] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican_service_user.split_loggers = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.793644] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] barbican_service_user.timeout = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.793828] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vault.approle_role_id = **** {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.794004] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vault.approle_secret_id = **** {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.794175] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vault.kv_mountpoint = secret {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.794787] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vault.kv_path = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.794787] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vault.kv_version = 2 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.794787] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vault.namespace = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.794787] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vault.root_token_id = **** {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.795135] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vault.ssl_ca_crt_file = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.795135] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vault.timeout = 60.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.795272] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vault.use_ssl = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.795422] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vault.vault_url = http://127.0.0.1:8200 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.795620] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] keystone.auth_section = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.795781] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] keystone.auth_type = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.795935] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] keystone.cafile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.796104] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] keystone.certfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.796265] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] keystone.collect_timing = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.796425] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] keystone.connect_retries = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.796580] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] keystone.connect_retry_delay = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.796734] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] keystone.endpoint_override = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.796890] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] keystone.insecure = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.797045] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] keystone.keyfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.797200] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] keystone.max_version = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.797351] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] keystone.min_version = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.797509] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] keystone.region_name = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.797663] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] keystone.retriable_status_codes = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.797816] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] keystone.service_name = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.797981] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] keystone.service_type = identity {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.798140] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] keystone.split_loggers = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.798294] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] keystone.status_code_retries = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.798449] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] keystone.status_code_retry_delay = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.798603] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] keystone.timeout = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.798775] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] keystone.valid_interfaces = ['internal', 'public'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.798930] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] keystone.version = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.799167] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.connection_uri = {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.799355] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.cpu_mode = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.799560] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.cpu_model_extra_flags = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.799730] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.cpu_models = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.799894] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.cpu_power_governor_high = performance {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.800071] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.cpu_power_governor_low = powersave {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.800237] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.cpu_power_management = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.800423] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.cpu_power_management_strategy = cpu_state {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.800604] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.device_detach_attempts = 8 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.800780] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.device_detach_timeout = 20 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.800943] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.disk_cachemodes = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.801097] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.disk_prefix = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.801257] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.enabled_perf_events = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.801414] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.file_backed_memory = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.801573] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.gid_maps = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.801748] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.hw_disk_discard = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.801904] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.hw_machine_type = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.802069] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.images_rbd_ceph_conf = {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.802234] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.images_rbd_glance_copy_poll_interval = 15 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.802397] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.images_rbd_glance_copy_timeout = 600 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.802561] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.images_rbd_glance_store_name = {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.802727] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.images_rbd_pool = rbd {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.802913] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.images_type = default {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.803074] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.images_volume_group = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.803253] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.inject_key = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.803417] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.inject_partition = -2 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.803576] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.inject_password = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.803735] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.iscsi_iface = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.803918] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.iser_use_multipath = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.804093] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.live_migration_bandwidth = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.804255] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.live_migration_completion_timeout = 800 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.804415] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.live_migration_downtime = 500 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.804572] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.live_migration_downtime_delay = 75 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.804728] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.live_migration_downtime_steps = 10 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.804899] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.live_migration_inbound_addr = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.805036] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.live_migration_permit_auto_converge = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.805193] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.live_migration_permit_post_copy = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.805348] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.live_migration_scheme = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.805514] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.live_migration_timeout_action = abort {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.805671] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.live_migration_tunnelled = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.805824] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.live_migration_uri = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.805981] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.live_migration_with_native_tls = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.806159] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.max_queues = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.806321] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.mem_stats_period_seconds = 10 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.806572] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.migration_inbound_addr = 10.180.1.21 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.806736] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.nfs_mount_options = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.806959] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.nfs_mount_point_base = /opt/stack/data/nova/mnt {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.807128] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.num_aoe_discover_tries = 3 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.807291] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.num_iser_scan_tries = 5 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.807447] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.num_memory_encrypted_guests = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.807606] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.num_nvme_discover_tries = 5 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.807766] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.num_pcie_ports = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.807927] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.num_volume_scan_tries = 5 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.808112] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.pmem_namespaces = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.808284] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.quobyte_client_cfg = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.808501] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.quobyte_mount_point_base = /opt/stack/data/nova/mnt {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.808663] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.rbd_connect_timeout = 5 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.808822] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.rbd_destroy_volume_retries = 12 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.808982] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.rbd_destroy_volume_retry_interval = 5 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.809135] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.rbd_secret_uuid = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.809303] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.rbd_user = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.809447] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.realtime_scheduler_priority = 1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.809636] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.remote_filesystem_transport = ssh {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.809793] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.rescue_image_id = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.809947] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.rescue_kernel_id = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.810101] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.rescue_ramdisk_id = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.810264] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.rng_dev_path = /dev/urandom {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.810418] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.rx_queue_size = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.810579] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.smbfs_mount_options = {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.810789] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.smbfs_mount_point_base = /opt/stack/data/nova/mnt {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.811001] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.snapshot_compression = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.811106] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.snapshot_image_format = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.811318] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.snapshots_directory = /opt/stack/data/nova/instances/snapshots {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.811479] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.sparse_logical_volumes = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.811642] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.swtpm_enabled = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.811826] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.swtpm_group = tss {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.811992] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.swtpm_user = tss {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.812175] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.sysinfo_serial = unique {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.812332] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.tb_cache_size = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.812489] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.tx_queue_size = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.812647] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.uid_maps = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.812804] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.use_virtio_for_bridges = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.812974] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.virt_type = kvm {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.813136] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.volume_clear = zero {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.813297] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.volume_clear_size = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.813456] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.volume_use_multipath = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.813610] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.vzstorage_cache_path = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.813772] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.vzstorage_log_path = /var/log/vstorage/%(cluster_name)s/nova.log.gz {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.813933] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.vzstorage_mount_group = qemu {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.814114] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.vzstorage_mount_opts = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.814278] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.vzstorage_mount_perms = 0770 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.814488] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.vzstorage_mount_point_base = /opt/stack/data/nova/mnt {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.814655] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.vzstorage_mount_user = stack {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.814820] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] libvirt.wait_soft_reboot_seconds = 120 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.815013] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.auth_section = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.815199] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.auth_type = password {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.815361] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.cafile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.815517] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.certfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.815676] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.collect_timing = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.815851] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.connect_retries = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.816016] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.connect_retry_delay = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.816212] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.default_floating_pool = public {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.816373] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.endpoint_override = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.816534] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.extension_sync_interval = 600 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.816691] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.http_retries = 3 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.816849] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.insecure = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.817002] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.keyfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.817158] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.max_version = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.817342] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.metadata_proxy_shared_secret = **** {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.817500] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.min_version = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.817662] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.ovs_bridge = br-int {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.817879] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.physnets = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.818046] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.region_name = RegionOne {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.818203] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.retriable_status_codes = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.818369] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.service_metadata_proxy = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.818524] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.service_name = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.818692] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.service_type = network {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.818849] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.split_loggers = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.819001] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.status_code_retries = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.819152] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.status_code_retry_delay = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.819307] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.timeout = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.819487] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.valid_interfaces = ['internal', 'public'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.819647] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] neutron.version = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.819814] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] notifications.bdms_in_notifications = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.819986] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] notifications.default_level = INFO {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.820169] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] notifications.notification_format = unversioned {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.820331] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] notifications.notify_on_state_change = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.820503] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] notifications.versioned_notifications_topics = ['versioned_notifications'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.820677] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] pci.alias = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.820840] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] pci.device_spec = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.821000] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] pci.report_in_placement = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.821206] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.auth_section = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.821381] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.auth_type = password {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.821562] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.auth_url = http://10.180.1.21/identity {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.821719] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.cafile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.821874] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.certfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.822030] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.collect_timing = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.822184] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.connect_retries = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.822337] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.connect_retry_delay = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.822492] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.default_domain_id = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.822644] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.default_domain_name = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.822815] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.domain_id = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.822969] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.domain_name = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.823124] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.endpoint_override = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.823283] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.insecure = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.823436] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.keyfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.823589] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.max_version = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.823740] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.min_version = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.824140] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.password = **** {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.824336] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.project_domain_id = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.824506] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.project_domain_name = Default {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.824669] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.project_id = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.824838] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.project_name = service {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.825007] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.region_name = RegionOne {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.825165] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.retriable_status_codes = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.825323] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.service_name = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.825493] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.service_type = placement {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.825651] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.split_loggers = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.825806] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.status_code_retries = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.825960] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.status_code_retry_delay = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.826113] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.system_scope = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.826268] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.timeout = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.826422] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.trust_id = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.826579] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.user_domain_id = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.826740] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.user_domain_name = Default {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.826896] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.user_id = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.827063] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.username = nova {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.827237] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.valid_interfaces = ['internal', 'public'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.827393] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] placement.version = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.827587] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] quota.cores = 20 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.827747] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] quota.count_usage_from_placement = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.827911] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] quota.driver = nova.quota.DbQuotaDriver {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.828106] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] quota.injected_file_content_bytes = 10240 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.828273] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] quota.injected_file_path_length = 255 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.828440] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] quota.injected_files = 5 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.828600] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] quota.instances = 10 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.828763] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] quota.key_pairs = 100 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.828924] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] quota.metadata_items = 128 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.829084] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] quota.ram = 51200 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.829251] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] quota.recheck_quota = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.829407] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] quota.server_group_members = 10 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.829566] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] quota.server_groups = 10 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.829765] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] scheduler.discover_hosts_in_cells_interval = -1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.829949] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] scheduler.enable_isolated_aggregate_filtering = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.830128] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] scheduler.image_metadata_prefilter = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.830292] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] scheduler.limit_tenants_to_placement_aggregate = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.830457] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] scheduler.max_attempts = 3 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.830615] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] scheduler.max_placement_results = 1000 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.830793] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] scheduler.placement_aggregate_required_for_tenants = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.830951] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] scheduler.query_placement_for_image_type_support = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.831192] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] scheduler.query_placement_for_routed_network_aggregates = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.831311] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] scheduler.workers = 2 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.831508] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.aggregate_image_properties_isolation_namespace = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.831674] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.aggregate_image_properties_isolation_separator = . {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.831866] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.available_filters = ['nova.scheduler.filters.all_filters'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.832043] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.build_failure_weight_multiplier = 1000000.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.832210] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.cpu_weight_multiplier = 1.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.832373] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.cross_cell_move_weight_multiplier = 1000000.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.832533] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.disk_weight_multiplier = 1.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.832737] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.enabled_filters = ['ComputeFilter', 'ComputeCapabilitiesFilter', 'ImagePropertiesFilter', 'ServerGroupAntiAffinityFilter', 'ServerGroupAffinityFilter', 'SameHostFilter', 'DifferentHostFilter'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.832908] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.host_subset_size = 1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.833072] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.hypervisor_version_weight_multiplier = 1.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.833247] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.image_properties_default_architecture = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.833409] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.io_ops_weight_multiplier = -1.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.833568] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.isolated_hosts = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.833724] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.isolated_images = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.833880] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.max_instances_per_host = 50 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.834035] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.max_io_ops_per_host = 8 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.834195] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.num_instances_weight_multiplier = 0.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.834354] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.pci_in_placement = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.834510] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.pci_weight_multiplier = 1.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.834666] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.ram_weight_multiplier = 1.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.834828] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.restrict_isolated_hosts_to_isolated_images = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.834985] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.shuffle_best_same_weighed_hosts = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.835142] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.soft_affinity_weight_multiplier = 1.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.835299] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.soft_anti_affinity_weight_multiplier = 1.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.835460] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.track_instance_changes = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.835631] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] filter_scheduler.weight_classes = ['nova.scheduler.weights.all_weighers'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.835801] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] metrics.required = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.835962] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] metrics.weight_multiplier = 1.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.836148] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] metrics.weight_of_unavailable = -10000.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.836313] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] metrics.weight_setting = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.836643] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] serial_console.base_url = ws://127.0.0.1:6083/ {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.836816] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] serial_console.enabled = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.837008] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] serial_console.port_range = 10000:20000 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.837178] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] serial_console.proxyclient_address = 127.0.0.1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.837341] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] serial_console.serialproxy_host = 0.0.0.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.837507] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] serial_console.serialproxy_port = 6083 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.837670] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] service_user.auth_section = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.837836] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] service_user.auth_type = password {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.837992] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] service_user.cafile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.838148] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] service_user.certfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.838302] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] service_user.collect_timing = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.838457] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] service_user.insecure = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.838609] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] service_user.keyfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.838771] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] service_user.send_service_user_token = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.838929] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] service_user.split_loggers = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.839082] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] service_user.timeout = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.839278] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] spice.agent_enabled = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.839430] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] spice.enabled = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.839758] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] spice.html5proxy_base_url = http://127.0.0.1:6082/spice_auto.html {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.839979] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] spice.html5proxy_host = 0.0.0.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.840162] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] spice.html5proxy_port = 6082 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.840355] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] spice.image_compression = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.840509] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] spice.jpeg_compression = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.840664] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] spice.playback_compression = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.840828] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] spice.server_listen = 127.0.0.1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.840992] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] spice.server_proxyclient_address = 127.0.0.1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.841144] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] spice.streaming_mode = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.841292] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] spice.zlib_compression = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.841474] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] upgrade_levels.baseapi = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.841644] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] upgrade_levels.compute = auto {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.841809] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] upgrade_levels.conductor = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.841969] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] upgrade_levels.scheduler = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.842133] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vendordata_dynamic_auth.auth_section = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.842291] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vendordata_dynamic_auth.auth_type = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.842446] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vendordata_dynamic_auth.cafile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.842598] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vendordata_dynamic_auth.certfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.842752] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vendordata_dynamic_auth.collect_timing = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.842905] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vendordata_dynamic_auth.insecure = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.843055] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vendordata_dynamic_auth.keyfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.843211] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vendordata_dynamic_auth.split_loggers = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.843366] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vendordata_dynamic_auth.timeout = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.843574] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.api_retry_count = 10 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.843729] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.ca_file = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.843882] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.cache_prefix = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.844054] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.cluster_name = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.844217] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.connection_pool_size = 10 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.844373] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.console_delay_seconds = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.844526] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.datastore_regex = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.844686] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.host_ip = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.844842] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.host_password = **** {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.845004] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.host_port = 443 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.845163] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.host_username = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.845339] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.insecure = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.845470] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.integration_bridge = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.845632] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.maximum_objects = 100 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.845786] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.pbm_default_policy = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.845946] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.pbm_enabled = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.846100] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.pbm_wsdl_location = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.846264] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.serial_log_dir = /opt/vmware/vspc {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.846419] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.serial_port_proxy_uri = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.846573] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.serial_port_service_uri = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.846733] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.task_poll_interval = 0.5 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.846890] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.use_linked_clone = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.847052] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.vnc_keymap = en-us {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.847219] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.vnc_port = 5900 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.847382] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vmware.vnc_port_total = 10000 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.847594] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vnc.auth_schemes = ['none'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.847757] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vnc.enabled = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.848068] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vnc.novncproxy_base_url = http://127.0.0.1:6080/vnc_auto.html {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.848251] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vnc.novncproxy_host = 0.0.0.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.848420] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vnc.novncproxy_port = 6080 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.848592] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vnc.server_listen = 127.0.0.1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.848758] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vnc.server_proxyclient_address = 127.0.0.1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.848914] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vnc.vencrypt_ca_certs = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.849072] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vnc.vencrypt_client_cert = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.849235] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] vnc.vencrypt_client_key = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.849433] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] workarounds.disable_compute_service_check_for_ffu = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.849596] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] workarounds.disable_deep_image_inspection = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.849753] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] workarounds.disable_fallback_pcpu_query = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.849906] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] workarounds.disable_group_policy_check_upcall = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.850061] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] workarounds.disable_libvirt_livesnapshot = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.850216] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] workarounds.disable_rootwrap = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.850371] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] workarounds.enable_numa_live_migration = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.850525] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] workarounds.enable_qemu_monitor_announce_self = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.850679] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] workarounds.ensure_libvirt_rbd_instance_dir_cleanup = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.850834] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] workarounds.handle_virt_lifecycle_events = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.850987] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] workarounds.libvirt_disable_apic = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.851143] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] workarounds.never_download_image_if_on_rbd = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.851299] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] workarounds.qemu_monitor_announce_self_count = 3 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.851456] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] workarounds.qemu_monitor_announce_self_interval = 1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.851617] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] workarounds.reserve_disk_resource_for_image_cache = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.851772] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] workarounds.skip_cpu_compare_at_startup = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.851927] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] workarounds.skip_cpu_compare_on_dest = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.852093] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] workarounds.skip_hypervisor_version_check_on_lm = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.852253] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] workarounds.skip_reserve_in_use_ironic_nodes = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.852410] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] workarounds.unified_limits_count_pcpu_as_vcpu = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.852569] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] workarounds.wait_for_vif_plugged_event_during_hard_reboot = [] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.852769] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] wsgi.api_paste_config = /etc/nova/api-paste.ini {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.852933] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] wsgi.client_socket_timeout = 900 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.853093] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] wsgi.default_pool_size = 1000 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.853253] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] wsgi.keep_alive = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.853414] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] wsgi.max_header_line = 16384 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.853570] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] wsgi.secure_proxy_ssl_header = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.853726] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] wsgi.ssl_ca_file = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.853879] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] wsgi.ssl_cert_file = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.854032] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] wsgi.ssl_key_file = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.854189] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] wsgi.tcp_keepidle = 600 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.854382] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] wsgi.wsgi_log_format = %(client_ip)s "%(request_line)s" status: %(status_code)s len: %(body_length)s time: %(wall_seconds).7f {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.854545] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] zvm.ca_file = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.854701] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] zvm.cloud_connector_url = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.854915] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] zvm.image_tmp_path = /opt/stack/data/nova/images {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.855080] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] zvm.reachable_timeout = 300 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.855298] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_policy.enforce_new_defaults = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.855475] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_policy.enforce_scope = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.855643] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_policy.policy_default_rule = default {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.855820] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.855987] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_policy.policy_file = policy.yaml {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.856196] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.856357] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.856512] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.856665] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.856822] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.857017] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.857216] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.857420] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] profiler.connection_string = messaging:// {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.857589] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] profiler.enabled = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.857769] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] profiler.es_doc_type = notification {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.857952] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] profiler.es_scroll_size = 10000 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.858116] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] profiler.es_scroll_time = 2m {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.858276] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] profiler.filter_error_trace = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.858439] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] profiler.hmac_keys = **** {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.858603] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] profiler.sentinel_service_name = mymaster {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.858763] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] profiler.socket_timeout = 0.1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.858920] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] profiler.trace_requests = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.859075] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] profiler.trace_sqlalchemy = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.859289] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] profiler_jaeger.process_tags = {} {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.859451] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] profiler_jaeger.service_name_prefix = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.859609] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] profiler_otlp.service_name_prefix = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.859798] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] remote_debug.host = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.859972] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] remote_debug.port = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.860174] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.860336] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.860506] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.860668] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.860830] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.860986] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.861148] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.heartbeat_in_pthread = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.861307] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.heartbeat_rate = 3 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.861486] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.861633] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.hostname = devstack {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.861790] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.861958] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.862122] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.862290] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.862457] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.processname = nova-conductor {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.862624] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.862783] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.862954] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.863113] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.863272] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.rabbit_quorum_delivery_limit = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.863434] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.rabbit_quorum_max_memory_bytes = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.863592] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.rabbit_quorum_max_memory_length = 0 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.863751] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.863912] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.864088] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.864250] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.rabbit_stream_fanout = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.864413] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.864569] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.rabbit_transient_quorum_queue = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.864733] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.864900] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.ssl = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.865089] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.865272] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.865432] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.ssl_enforce_fips_mode = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.865602] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.865767] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.ssl_version = {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.865924] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_rabbit.use_queue_manager = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.866142] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_notifications.driver = ['messagingv2'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.866313] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_notifications.retry = -1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.866491] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.866660] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_messaging_notifications.transport_url = **** {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.866860] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_limit.auth_section = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.867025] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_limit.auth_type = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.867182] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_limit.cafile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.867333] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_limit.certfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.867494] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_limit.collect_timing = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.867649] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_limit.connect_retries = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.867802] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_limit.connect_retry_delay = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.867954] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_limit.endpoint_id = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.868122] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_limit.endpoint_override = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.868280] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_limit.insecure = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.868437] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_limit.keyfile = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.868589] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_limit.max_version = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.868742] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_limit.min_version = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.868892] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_limit.region_name = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.869046] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_limit.retriable_status_codes = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.869226] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_limit.service_name = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.869364] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_limit.service_type = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.869518] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_limit.split_loggers = False {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.869670] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_limit.status_code_retries = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.869824] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_limit.status_code_retry_delay = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.869980] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_limit.timeout = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.870157] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_limit.valid_interfaces = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.870310] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_limit.version = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.870514] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_reports.file_event_handler = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.870676] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_reports.file_event_handler_interval = 1 {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.870828] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] oslo_reports.log_dir = None {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 392.870953] nova-conductor[54261]: DEBUG oslo_service.service [None req-79af25d7-0948-46e9-830e-047783347c91 None None] ******************************************************************************** {{(pid=54261) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2624}} [ 402.620085] nova-conductor[54261]: DEBUG dbcounter [-] [54261] Writing DB stats nova_api:SELECT=1 {{(pid=54261) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 402.710338] nova-conductor[54261]: DEBUG dbcounter [-] [54261] Writing DB stats nova_cell0:SELECT=1 {{(pid=54261) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 402.710338] nova-conductor[54261]: DEBUG dbcounter [-] [54261] Writing DB stats nova_cell1:SELECT=1 {{(pid=54261) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 407.790879] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:SELECT=4,nova_cell1:INSERT=1,nova_cell1:UPDATE=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 407.798376] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=5,nova_cell1:INSERT=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 479.329531] nova-conductor[54692]: DEBUG oslo_db.sqlalchemy.engines [None req-eb94dca2-381c-4764-94da-37038f29e7d7 None None] Parent process 54261 forked (54692) with an open database connection, which is being discarded and recreated. {{(pid=54692) checkout /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:438}} [ 479.331456] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writer thread running {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} [ 489.332221] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 496.473013] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:SELECT=7,nova_cell1:INSERT=1,nova_cell1:UPDATE=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 501.986283] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=7,nova_cell1:INSERT=1,nova_cell1:UPDATE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 523.410492] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Creating reply queue: reply_be4b6a8028724e64b8a3a40877a630ee [ 523.418165] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Expecting reply to msg 6bb8b018024047d49133ad1020f729e9 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 524.265450] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 6bb8b018024047d49133ad1020f729e9 [ 524.422923] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Took 1.02 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 524.428035] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=2 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 524.449457] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 524.451974] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.003s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 524.453645] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.002s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 524.455663] nova-conductor[54692]: INFO dbcounter [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Registered counter for database nova_cell1 [ 524.458755] nova-conductor[54692]: DEBUG oslo_db.sqlalchemy.engines [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=54692) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:342}} [ 524.465186] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writer thread running {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} [ 524.533075] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 524.533308] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 524.533734] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 524.534147] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 524.534334] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 524.534497] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 524.536059] nova-conductor[54692]: INFO dbcounter [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Registered counter for database nova_cell0 [ 524.543641] nova-conductor[54692]: DEBUG oslo_db.sqlalchemy.engines [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=54692) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:342}} [ 524.550154] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writer thread running {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} [ 524.560525] nova-conductor[54692]: DEBUG nova.quota [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Getting quotas for project de11ac365f704c2986c782a5918f802f. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 524.565009] nova-conductor[54692]: DEBUG nova.quota [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Getting quotas for user c43c33165931465494c0a7da171126e4 and project de11ac365f704c2986c782a5918f802f. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 524.569948] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: 01127e18-3d68-4b50-b0d4-89f186aa7fb4] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 524.570630] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 524.570909] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 524.571157] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 524.585961] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: 01127e18-3d68-4b50-b0d4-89f186aa7fb4] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 524.586787] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 524.587040] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 524.587266] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 524.624430] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 524.624854] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 524.625197] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 524.625607] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "compute-rpcapi-router" {{(pid=54692) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 524.625963] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquired lock "compute-rpcapi-router" {{(pid=54692) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 524.626626] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fc830098-2a4f-4d61-94f6-e132043077ec None None] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 524.626985] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fc830098-2a4f-4d61-94f6-e132043077ec None None] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 524.631444] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fc830098-2a4f-4d61-94f6-e132043077ec None None] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.003s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 524.631444] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fc830098-2a4f-4d61-94f6-e132043077ec None None] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 524.631444] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fc830098-2a4f-4d61-94f6-e132043077ec None None] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 524.631444] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fc830098-2a4f-4d61-94f6-e132043077ec None None] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 524.643235] nova-conductor[54692]: INFO nova.compute.rpcapi [None req-fc830098-2a4f-4d61-94f6-e132043077ec None None] Automatically selected compute RPC version 6.3 from minimum service version 67 [ 524.643781] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fc830098-2a4f-4d61-94f6-e132043077ec None None] Releasing lock "compute-rpcapi-router" {{(pid=54692) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 524.837702] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Expecting reply to msg 6c5a9860bc564a3aa03a28fa7897eab4 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 525.105786] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 6c5a9860bc564a3aa03a28fa7897eab4 [ 525.352123] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Took 0.52 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 525.366710] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 525.366824] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 525.366973] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 525.420173] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 525.420627] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 525.420947] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 525.422001] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 525.422333] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 525.422660] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 525.433211] nova-conductor[54692]: DEBUG nova.quota [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Getting quotas for project f9f409d646e3429cac4cd3555a5073c2. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 525.437992] nova-conductor[54692]: DEBUG nova.quota [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Getting quotas for user a9a0e0bbae8046a783b171c8a4ee3c11 and project f9f409d646e3429cac4cd3555a5073c2. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 525.443295] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] [instance: d995b3ce-c5b5-490e-b774-706a74632f27] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 525.444042] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 525.444403] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 525.444717] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 525.448627] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] [instance: d995b3ce-c5b5-490e-b774-706a74632f27] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 525.450039] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 525.450441] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 525.450913] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 525.487689] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 525.488081] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 525.488359] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 525.616182] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writer thread running {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} [ 525.624450] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Creating reply queue: reply_4316ed433d59463c9660e50df524570b [ 525.631380] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Expecting reply to msg 9b39ed238c174a3bbf9cd85efd396a88 in queue reply_4316ed433d59463c9660e50df524570b [ 525.800908] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 9b39ed238c174a3bbf9cd85efd396a88 [ 526.076958] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Expecting reply to msg f177d4e29ebf4469bfea83648110add3 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 526.134990] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 526.141417] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=2 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 526.167513] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 526.167806] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 526.169662] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.002s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 526.171661] nova-conductor[54691]: INFO dbcounter [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Registered counter for database nova_cell1 [ 526.175439] nova-conductor[54691]: DEBUG oslo_db.sqlalchemy.engines [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=54691) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:342}} [ 526.182308] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writer thread running {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} [ 526.251046] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Expecting reply to msg 8379a95ad154444f8a3cb857d714f9e4 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 526.268165] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg f177d4e29ebf4469bfea83648110add3 [ 526.275503] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 526.275740] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 526.276291] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 526.276731] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 526.276926] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 526.277501] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 526.278687] nova-conductor[54691]: INFO dbcounter [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Registered counter for database nova_cell0 [ 526.285520] nova-conductor[54691]: DEBUG oslo_db.sqlalchemy.engines [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=54691) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:342}} [ 526.289959] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writer thread running {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} [ 526.297981] nova-conductor[54691]: DEBUG nova.quota [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Getting quotas for project f9b8a0b9692647368b3e08b6219291de. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 526.301233] nova-conductor[54691]: DEBUG nova.quota [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Getting quotas for user 0e2b5df479b642c68c88c8608359ea4d and project f9b8a0b9692647368b3e08b6219291de. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 526.306421] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] [instance: 03b95ad5-85fe-41ad-987f-8140380f0a0d] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 526.306996] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 526.307251] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 526.307339] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 526.313837] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] [instance: 03b95ad5-85fe-41ad-987f-8140380f0a0d] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 526.314572] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 526.314754] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 526.314919] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 526.342260] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 526.342520] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 526.342693] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 526.342948] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Acquiring lock "compute-rpcapi-router" {{(pid=54691) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 526.343087] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Acquired lock "compute-rpcapi-router" {{(pid=54691) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 526.344226] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ab80f71-7052-4315-b05f-f5d2418748b8 None None] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 526.344226] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ab80f71-7052-4315-b05f-f5d2418748b8 None None] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 526.344374] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ab80f71-7052-4315-b05f-f5d2418748b8 None None] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 526.344679] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ab80f71-7052-4315-b05f-f5d2418748b8 None None] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 526.345082] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ab80f71-7052-4315-b05f-f5d2418748b8 None None] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 526.345384] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ab80f71-7052-4315-b05f-f5d2418748b8 None None] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 526.359276] nova-conductor[54691]: INFO nova.compute.rpcapi [None req-3ab80f71-7052-4315-b05f-f5d2418748b8 None None] Automatically selected compute RPC version 6.3 from minimum service version 67 [ 526.359868] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ab80f71-7052-4315-b05f-f5d2418748b8 None None] Releasing lock "compute-rpcapi-router" {{(pid=54691) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 526.398392] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 8379a95ad154444f8a3cb857d714f9e4 [ 526.579516] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 526.594457] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 526.594739] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 526.594912] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 526.638990] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 526.639217] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 526.639439] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 526.639830] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 526.640039] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 526.640205] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 526.651450] nova-conductor[54692]: DEBUG nova.quota [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Getting quotas for project 0c8a7ccd195e42f2a29370a2f89c9c67. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 526.654688] nova-conductor[54692]: DEBUG nova.quota [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Getting quotas for user 418f4c924f4d45e69a85395d28c6f8b8 and project 0c8a7ccd195e42f2a29370a2f89c9c67. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 526.660568] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] [instance: 1b8498b4-5186-4c0e-9b0b-9b8812e8a2e2] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 526.661134] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 526.661354] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 526.661520] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 526.665898] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] [instance: 1b8498b4-5186-4c0e-9b0b-9b8812e8a2e2] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 526.666896] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 526.667116] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 526.667388] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 526.696993] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 526.697225] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 526.697425] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 526.760167] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 526.785727] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 526.785727] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 526.785727] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 526.845226] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 526.845458] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 526.845639] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 526.846033] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 526.846234] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 526.846488] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 526.857568] nova-conductor[54692]: DEBUG nova.quota [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Getting quotas for project 77a894bddd55484d956e0569e6d26d38. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 526.861235] nova-conductor[54692]: DEBUG nova.quota [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Getting quotas for user 87f04a89778e4350a90bcc998a96de0f and project 77a894bddd55484d956e0569e6d26d38. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 526.871944] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] [instance: 2eb46a2a-27fc-4d44-9d18-6d4553026a60] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 526.872497] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 526.872702] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 526.872872] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 526.877801] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] [instance: 2eb46a2a-27fc-4d44-9d18-6d4553026a60] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 526.878480] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 526.878687] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 526.878858] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 526.903017] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 526.903242] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 526.903407] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 527.070363] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] Expecting reply to msg 191bf38f740249109d46acb276a59c49 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 527.244588] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 191bf38f740249109d46acb276a59c49 [ 527.574237] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 527.588368] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 527.588655] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 527.588893] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 527.622702] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 527.622957] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 527.623332] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 527.623545] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 527.623711] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 527.623891] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 527.634061] nova-conductor[54692]: DEBUG nova.quota [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] Getting quotas for project 1df47f750ca24959bc88e535e1c8f075. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 527.636426] nova-conductor[54692]: DEBUG nova.quota [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] Getting quotas for user ef4e9fa18a354f99a381e73b5cd03666 and project 1df47f750ca24959bc88e535e1c8f075. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 527.641716] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] [instance: 5228dcb4-ec4b-4cf3-b0cb-1f53adde9b91] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 527.642222] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 527.642454] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 527.642598] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 527.679448] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] [instance: 5228dcb4-ec4b-4cf3-b0cb-1f53adde9b91] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 527.679827] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 527.679984] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 527.680184] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 527.704836] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 527.705015] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 527.705182] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6983c055-f4fc-40ff-af3d-b1c177f65e24 tempest-ServerDiagnosticsV248Test-2096702815 tempest-ServerDiagnosticsV248Test-2096702815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 528.248437] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Expecting reply to msg c7e8588eee554325b7cfe5e8e56d77c8 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 528.414523] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg c7e8588eee554325b7cfe5e8e56d77c8 [ 528.759837] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Took 0.52 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 528.775557] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 528.775781] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 528.775948] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 528.815498] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 528.815752] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 528.815927] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 528.816352] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 528.816607] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 528.816708] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 528.825807] nova-conductor[54692]: DEBUG nova.quota [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Getting quotas for project 6ce3b399d3c649618af0af533baaf84b. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 528.827810] nova-conductor[54692]: DEBUG nova.quota [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Getting quotas for user a145ce1c57cd43379c748e2731e3865c and project 6ce3b399d3c649618af0af533baaf84b. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 528.832333] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] [instance: 0b211be4-1955-456d-bf30-5195b5169807] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 528.832833] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 528.833032] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 528.833195] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 528.841394] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] [instance: 0b211be4-1955-456d-bf30-5195b5169807] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 528.842157] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 528.842306] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 528.842468] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 528.859752] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 528.859986] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 528.860116] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 528.973878] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Expecting reply to msg 2f817ef878e04677bb06b4ad573ae856 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 529.135451] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 2f817ef878e04677bb06b4ad573ae856 [ 529.484207] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 529.516807] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 529.517044] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 529.517293] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 529.552258] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 529.552481] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 529.553864] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 529.553864] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 529.554034] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 529.554193] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 529.562486] nova-conductor[54692]: DEBUG nova.quota [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Getting quotas for project f9f409d646e3429cac4cd3555a5073c2. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 529.564529] nova-conductor[54692]: DEBUG nova.quota [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Getting quotas for user a9a0e0bbae8046a783b171c8a4ee3c11 and project f9f409d646e3429cac4cd3555a5073c2. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 529.569639] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] [instance: ae1041c6-888a-4cb3-8da3-00e43b5b6531] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 529.570055] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 529.570319] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 529.570696] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 529.575969] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] [instance: ae1041c6-888a-4cb3-8da3-00e43b5b6531] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 529.577172] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 529.577172] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 529.577172] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 529.593855] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 529.594016] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 529.594188] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 529.925652] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Expecting reply to msg 9d71094c361b4f96a47197b087ce7d43 in queue reply_4316ed433d59463c9660e50df524570b [ 530.085961] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 9d71094c361b4f96a47197b087ce7d43 [ 530.430142] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 530.442746] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 530.442988] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 530.443214] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 530.484977] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 530.485215] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 530.485385] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 530.485751] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 530.485951] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 530.486113] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 530.496456] nova-conductor[54691]: DEBUG nova.quota [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Getting quotas for project 204d8f668a894827994c416d74f34c18. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 530.498919] nova-conductor[54691]: DEBUG nova.quota [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Getting quotas for user ea7e09048589421e96a4d19f8e884c8a and project 204d8f668a894827994c416d74f34c18. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 530.505146] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: aa9671f3-07a8-422f-93b5-006de0f2cb3c] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 530.505730] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 530.505945] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 530.506114] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 530.511172] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: aa9671f3-07a8-422f-93b5-006de0f2cb3c] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 530.512138] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 530.512366] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 530.512568] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 530.533719] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 530.534006] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 530.534193] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 530.818923] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Expecting reply to msg f01510aaa12c481b88509c3c72867e8f in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 531.012862] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg f01510aaa12c481b88509c3c72867e8f [ 531.321988] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 531.335861] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 531.336119] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 531.336297] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 531.369241] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 531.369706] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 531.369918] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 531.370296] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 531.370512] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 531.370676] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 531.380452] nova-conductor[54692]: DEBUG nova.quota [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Getting quotas for project 7f30e3b24f414007b0066cab5346a3bf. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 531.382631] nova-conductor[54692]: DEBUG nova.quota [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Getting quotas for user 71886e19fbc749d989c55e17c178862a and project 7f30e3b24f414007b0066cab5346a3bf. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 531.387572] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] [instance: a5a690d3-8018-4cc5-8d59-55062a595ed0] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 531.388185] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 531.388441] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 531.388665] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 531.394404] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] [instance: a5a690d3-8018-4cc5-8d59-55062a595ed0] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 531.395890] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 531.395890] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 531.395890] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 531.421262] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 531.421262] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 531.421620] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 531.643246] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Expecting reply to msg ce718ee0e6934c3eb552bc53d7444017 in queue reply_4316ed433d59463c9660e50df524570b [ 531.782626] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg ce718ee0e6934c3eb552bc53d7444017 [ 532.146097] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 532.160457] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 532.160457] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 532.160457] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 532.195314] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 532.195514] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 532.195689] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 532.196309] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 532.196381] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 532.196757] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 532.224515] nova-conductor[54691]: DEBUG nova.quota [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Getting quotas for project f9f409d646e3429cac4cd3555a5073c2. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 532.226610] nova-conductor[54691]: DEBUG nova.quota [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Getting quotas for user a9a0e0bbae8046a783b171c8a4ee3c11 and project f9f409d646e3429cac4cd3555a5073c2. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 532.232534] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] [instance: 01b348db-6a0d-4283-b9d9-8edacfd1c219] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 532.233164] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 532.233441] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 532.233767] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 532.238981] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] [instance: 01b348db-6a0d-4283-b9d9-8edacfd1c219] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 532.239816] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 532.240115] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 532.240356] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 532.274871] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 532.275130] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 532.275324] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 533.811262] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Expecting reply to msg 0ea2c86b77294c8c8f79f6484a6d964f in queue reply_4316ed433d59463c9660e50df524570b [ 534.066984] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 0ea2c86b77294c8c8f79f6484a6d964f [ 534.313805] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 534.328807] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 534.329043] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 534.329212] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 534.368927] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 534.369198] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 534.369403] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 534.369768] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 534.370070] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 534.370138] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 534.392452] nova-conductor[54691]: DEBUG nova.quota [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Getting quotas for project 203ab2a2f81f449e81ec2fdeb589380a. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 534.395416] nova-conductor[54691]: DEBUG nova.quota [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Getting quotas for user 6132ab51d5b34a8c9c21a6aa642bc15d and project 203ab2a2f81f449e81ec2fdeb589380a. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 534.405709] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] [instance: 958e85c0-cb75-4dfd-92c4-b68957e87ec5] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 534.406584] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 534.406918] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 534.407092] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 534.412961] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] [instance: 958e85c0-cb75-4dfd-92c4-b68957e87ec5] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 534.413964] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 534.414173] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 534.414337] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 534.438630] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 534.438681] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 534.438819] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 536.211110] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Expecting reply to msg 664be0417a1346d68000c3c2ee36f527 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 536.380058] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 664be0417a1346d68000c3c2ee36f527 [ 536.714126] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 536.728645] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 536.728884] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 536.729055] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 536.808229] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 536.810320] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.002s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 536.810582] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 536.811194] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 536.811194] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 536.811371] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 536.832785] nova-conductor[54692]: DEBUG nova.quota [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Getting quotas for project bc7ba68e037a4f28b05c14e7bdbb24e4. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 536.843180] nova-conductor[54692]: DEBUG nova.quota [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Getting quotas for user a5b16af3d328480fae79eae1aff63f64 and project bc7ba68e037a4f28b05c14e7bdbb24e4. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 536.852729] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] [instance: 52f8e827-3faf-47c1-b793-ccacc5a0bc4c] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 536.853371] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 536.853900] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 536.853900] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 536.861079] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] [instance: 52f8e827-3faf-47c1-b793-ccacc5a0bc4c] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 536.861414] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 536.861666] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 536.861858] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 536.907357] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 536.907859] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 536.908396] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 538.033058] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Expecting reply to msg e15d09f103eb481b91e492788a9d62c5 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 538.210732] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg e15d09f103eb481b91e492788a9d62c5 [ 538.535701] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 538.552562] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 538.552870] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 538.553264] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 538.592406] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 538.592768] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 538.592996] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 538.593433] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 538.593659] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 538.593858] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 538.607010] nova-conductor[54692]: DEBUG nova.quota [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Getting quotas for project 51d621f5cd014007a7d2e44bcd714cdf. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 538.615955] nova-conductor[54692]: DEBUG nova.quota [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Getting quotas for user 064e7b872e9a4d7f9b13a6b722121734 and project 51d621f5cd014007a7d2e44bcd714cdf. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 538.628069] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] [instance: f537a3a9-b63a-4233-968a-942aca6ada0e] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 538.628423] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 538.628814] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 538.628909] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 538.636708] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] [instance: f537a3a9-b63a-4233-968a-942aca6ada0e] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 538.637532] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 538.637779] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 538.637923] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 538.667169] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 538.667426] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 538.667634] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 541.615606] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Expecting reply to msg 94c89a0c274d4aa1b949e2f8e368daf0 in queue reply_4316ed433d59463c9660e50df524570b [ 541.802037] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 94c89a0c274d4aa1b949e2f8e368daf0 [ 542.118470] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 542.131662] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 542.131894] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 542.132080] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 542.165372] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 542.165636] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 542.165803] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 542.166179] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 542.166376] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 542.166551] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 542.175156] nova-conductor[54691]: DEBUG nova.quota [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Getting quotas for project f778f88dc6e94351ad33c2d0cf875b3b. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 542.178217] nova-conductor[54691]: DEBUG nova.quota [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Getting quotas for user 1d7d23e8e47c4620ba5a2d0de6d90280 and project f778f88dc6e94351ad33c2d0cf875b3b. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 542.184418] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] [instance: bd41759b-5c32-42e6-b625-4ea48f98045e] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 542.185019] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 542.185339] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 542.185618] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 542.189523] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] [instance: bd41759b-5c32-42e6-b625-4ea48f98045e] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 542.190356] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 542.190678] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 542.190955] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 542.211634] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 542.211990] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 542.212301] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 542.306991] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Expecting reply to msg a217150b3e19482197335f0f5b2098ff in queue reply_4316ed433d59463c9660e50df524570b [ 542.446566] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg a217150b3e19482197335f0f5b2098ff [ 542.817604] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 542.843737] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 542.844172] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 542.844424] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 542.884397] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 542.884629] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 542.884864] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 542.885755] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 542.886365] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 542.886562] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 542.901331] nova-conductor[54691]: DEBUG nova.quota [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Getting quotas for project 167ea2771f59412fbbb9ab0f2de63f68. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 542.907513] nova-conductor[54691]: DEBUG nova.quota [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Getting quotas for user 0df82cc3b5714a2389ac2a421876763f and project 167ea2771f59412fbbb9ab0f2de63f68. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 542.922520] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] [instance: 25508ff0-bc62-47b3-ae9e-7d9d8c735edc] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 542.922520] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 542.922520] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 542.922520] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 542.933168] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] [instance: 25508ff0-bc62-47b3-ae9e-7d9d8c735edc] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 542.934377] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 542.934876] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 542.935086] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 542.957371] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 542.957752] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 542.957815] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 547.607431] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Expecting reply to msg 18c649cd5b254cdaada89244a8327039 in queue reply_4316ed433d59463c9660e50df524570b [ 547.843171] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 18c649cd5b254cdaada89244a8327039 [ 548.110771] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 548.129417] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 548.129930] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 548.130234] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 548.174284] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 548.174513] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 548.174683] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 548.175046] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 548.175221] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 548.175385] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 548.185505] nova-conductor[54691]: DEBUG nova.quota [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Getting quotas for project 761a5ab005a34ef2b788c40b495432d6. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 548.188665] nova-conductor[54691]: DEBUG nova.quota [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Getting quotas for user 16deda22cf4d47a6af9d8c18533df8cd and project 761a5ab005a34ef2b788c40b495432d6. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 548.194599] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] [instance: 918aa256-a10d-46d1-b9a4-ecd70cf3566f] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 548.195201] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 548.195405] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 548.195605] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 548.200210] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] [instance: 918aa256-a10d-46d1-b9a4-ecd70cf3566f] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 548.201398] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 548.201612] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 548.202573] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 548.273562] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 548.273562] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 548.273786] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 548.598261] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell0:SELECT=11 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 548.640571] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=32,nova_cell1:SAVEPOINT=9,nova_cell1:INSERT=80,nova_cell1:RELEASE=9 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 548.651807] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=54,nova_api:UPDATE=5,nova_api:DELETE=6 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 548.654331] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=65,nova_api:UPDATE=5,nova_api:DELETE=4 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 548.832935] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Expecting reply to msg 387c199e383c4e21a1250c3528850b4f in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 548.985837] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 387c199e383c4e21a1250c3528850b4f [ 549.336148] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 549.352482] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 549.352721] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 549.352894] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 549.385907] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 549.386136] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 549.386304] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 549.386710] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 549.386901] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 549.387061] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 549.396236] nova-conductor[54692]: DEBUG nova.quota [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Getting quotas for project f1f2d9e07c344faf9fd688396711d718. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 549.399075] nova-conductor[54692]: DEBUG nova.quota [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Getting quotas for user d3df5db771204330838b73899c4646d0 and project f1f2d9e07c344faf9fd688396711d718. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 549.403997] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] [instance: 64ee370c-6d43-4e2b-a27a-3aa716fdc322] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 549.404542] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 549.404779] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 549.405059] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 549.409210] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] [instance: 64ee370c-6d43-4e2b-a27a-3aa716fdc322] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 549.409938] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 549.410167] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 549.410345] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 549.426251] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 549.426483] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 549.426656] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 554.448820] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] [instance: d995b3ce-c5b5-490e-b774-706a74632f27] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port fdbabc59-acd2-490d-95ce-d13232dc4eb2, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance d995b3ce-c5b5-490e-b774-706a74632f27 was re-scheduled: Binding failed for port fdbabc59-acd2-490d-95ce-d13232dc4eb2, please check neutron logs for more information.\n'] [ 554.449497] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 554.449782] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d995b3ce-c5b5-490e-b774-706a74632f27.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d995b3ce-c5b5-490e-b774-706a74632f27. [ 554.450224] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] [instance: d995b3ce-c5b5-490e-b774-706a74632f27] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d995b3ce-c5b5-490e-b774-706a74632f27. [ 554.514224] nova-conductor[54692]: DEBUG nova.network.neutron [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] [instance: d995b3ce-c5b5-490e-b774-706a74632f27] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 554.666069] nova-conductor[54692]: DEBUG nova.network.neutron [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] [instance: d995b3ce-c5b5-490e-b774-706a74632f27] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 554.671755] nova-conductor[54692]: DEBUG nova.network.neutron [None req-2cb5fb93-4f7d-4a5a-970c-db1b2fe4740e tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] [instance: d995b3ce-c5b5-490e-b774-706a74632f27] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 554.749412] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Expecting reply to msg 23bdb5776aa946dabb0d108d0348b440 in queue reply_4316ed433d59463c9660e50df524570b [ 554.915546] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 23bdb5776aa946dabb0d108d0348b440 [ 555.259613] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 555.277649] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 555.277649] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 555.277649] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 555.316791] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 555.317019] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 555.317202] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 555.317578] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 555.317757] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 555.317916] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 555.330908] nova-conductor[54691]: DEBUG nova.quota [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Getting quotas for project bc21137f57474824a35a5f0926cc04ef. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 555.337220] nova-conductor[54691]: DEBUG nova.quota [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Getting quotas for user 283bddf95dd940438dacbce6ef1d3733 and project bc21137f57474824a35a5f0926cc04ef. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 555.342215] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] [instance: 3a848010-382b-4753-8420-37222963e7e2] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 555.342720] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 555.342921] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 555.343082] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 555.350191] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] [instance: 3a848010-382b-4753-8420-37222963e7e2] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 555.350968] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 555.351185] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 555.351355] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 555.368231] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 555.368451] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 555.368613] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 555.709595] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Expecting reply to msg d105a75bdbaa4ce2b4fe78072a762f08 in queue reply_4316ed433d59463c9660e50df524570b [ 555.895473] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg d105a75bdbaa4ce2b4fe78072a762f08 [ 556.035785] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: 01127e18-3d68-4b50-b0d4-89f186aa7fb4] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port a989424f-fd97-4857-9982-210001ac2adc, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 01127e18-3d68-4b50-b0d4-89f186aa7fb4 was re-scheduled: Binding failed for port a989424f-fd97-4857-9982-210001ac2adc, please check neutron logs for more information.\n'] [ 556.035785] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 556.036062] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 01127e18-3d68-4b50-b0d4-89f186aa7fb4.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 01127e18-3d68-4b50-b0d4-89f186aa7fb4. [ 556.036434] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: 01127e18-3d68-4b50-b0d4-89f186aa7fb4] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 01127e18-3d68-4b50-b0d4-89f186aa7fb4. [ 556.085734] nova-conductor[54691]: DEBUG nova.network.neutron [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: 01127e18-3d68-4b50-b0d4-89f186aa7fb4] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 556.217721] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 556.233002] nova-conductor[54691]: DEBUG nova.network.neutron [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: 01127e18-3d68-4b50-b0d4-89f186aa7fb4] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 556.236332] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 556.236564] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 556.236734] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 556.242019] nova-conductor[54691]: DEBUG nova.network.neutron [None req-a3b85042-ae47-49c2-846d-542ff6197062 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: 01127e18-3d68-4b50-b0d4-89f186aa7fb4] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 556.279510] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 556.279749] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 556.279901] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 556.280303] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 556.280494] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 556.280678] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 556.288740] nova-conductor[54691]: DEBUG nova.quota [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Getting quotas for project c4c8d1a186584c33a1f71c1239b6d670. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 556.291826] nova-conductor[54691]: DEBUG nova.quota [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Getting quotas for user 4f17ce167bc14a11977fcecdf1420e7d and project c4c8d1a186584c33a1f71c1239b6d670. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 556.296541] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] [instance: c7670280-b5a7-4de2-8cf9-6814a81eb1a6] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 556.297021] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 556.297199] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 556.297369] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 556.300702] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] [instance: c7670280-b5a7-4de2-8cf9-6814a81eb1a6] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 556.301353] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 556.301549] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 556.301765] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 556.317523] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 556.317737] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 556.317900] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 557.784037] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] [instance: 03b95ad5-85fe-41ad-987f-8140380f0a0d] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port b10f2a3c-b187-4aa0-9d8d-5fbcf18d8c1f, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 03b95ad5-85fe-41ad-987f-8140380f0a0d was re-scheduled: Binding failed for port b10f2a3c-b187-4aa0-9d8d-5fbcf18d8c1f, please check neutron logs for more information.\n'] [ 557.784593] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 557.784811] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 03b95ad5-85fe-41ad-987f-8140380f0a0d.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 03b95ad5-85fe-41ad-987f-8140380f0a0d. [ 557.785131] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] [instance: 03b95ad5-85fe-41ad-987f-8140380f0a0d] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 03b95ad5-85fe-41ad-987f-8140380f0a0d. [ 557.816564] nova-conductor[54692]: DEBUG nova.network.neutron [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] [instance: 03b95ad5-85fe-41ad-987f-8140380f0a0d] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 557.832420] nova-conductor[54692]: DEBUG nova.network.neutron [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] [instance: 03b95ad5-85fe-41ad-987f-8140380f0a0d] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 557.835774] nova-conductor[54692]: DEBUG nova.network.neutron [None req-49aff994-c961-441e-80b7-68437e3a1ee8 tempest-ServerTagsTestJSON-682251749 tempest-ServerTagsTestJSON-682251749-project-member] [instance: 03b95ad5-85fe-41ad-987f-8140380f0a0d] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 559.326128] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Expecting reply to msg f4e6ba8a384b4f048711e63a087fee3a in queue reply_4316ed433d59463c9660e50df524570b [ 559.391406] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell0:SELECT=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 559.411958] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=3,nova_cell1:SAVEPOINT=1,nova_cell1:INSERT=8,nova_cell1:RELEASE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 559.420134] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=6,nova_api:UPDATE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 559.423720] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=6,nova_api:DELETE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 559.458243] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg f4e6ba8a384b4f048711e63a087fee3a [ 559.828889] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 559.841415] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 559.841669] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 559.841854] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 559.871592] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] [instance: 1b8498b4-5186-4c0e-9b0b-9b8812e8a2e2] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port f558dc42-a489-4138-8f8c-cfef1c9c730b, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 1b8498b4-5186-4c0e-9b0b-9b8812e8a2e2 was re-scheduled: Binding failed for port f558dc42-a489-4138-8f8c-cfef1c9c730b, please check neutron logs for more information.\n'] [ 559.872336] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 559.872633] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1b8498b4-5186-4c0e-9b0b-9b8812e8a2e2.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1b8498b4-5186-4c0e-9b0b-9b8812e8a2e2. [ 559.872888] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] [instance: 1b8498b4-5186-4c0e-9b0b-9b8812e8a2e2] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1b8498b4-5186-4c0e-9b0b-9b8812e8a2e2. [ 559.880652] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 559.880886] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 559.881059] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 559.881422] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 559.881635] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 559.881801] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 559.893334] nova-conductor[54691]: DEBUG nova.quota [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Getting quotas for project de11ac365f704c2986c782a5918f802f. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 559.895580] nova-conductor[54691]: DEBUG nova.quota [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Getting quotas for user c43c33165931465494c0a7da171126e4 and project de11ac365f704c2986c782a5918f802f. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 559.899801] nova-conductor[54692]: DEBUG nova.network.neutron [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] [instance: 1b8498b4-5186-4c0e-9b0b-9b8812e8a2e2] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 559.900661] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: 558058e0-9808-415a-8866-75d1c8598d43] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 559.901220] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 559.901467] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 559.901604] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 559.908107] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: 558058e0-9808-415a-8866-75d1c8598d43] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 559.908794] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 559.908993] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 559.909304] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 559.915561] nova-conductor[54692]: DEBUG nova.network.neutron [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] [instance: 1b8498b4-5186-4c0e-9b0b-9b8812e8a2e2] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 559.918855] nova-conductor[54692]: DEBUG nova.network.neutron [None req-7a66ceda-251c-4951-be29-f56f4cbbc493 tempest-ServerDiagnosticsNegativeTest-1606981752 tempest-ServerDiagnosticsNegativeTest-1606981752-project-member] [instance: 1b8498b4-5186-4c0e-9b0b-9b8812e8a2e2] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 559.939774] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 559.943599] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.004s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 559.943800] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 561.789694] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] [instance: 2eb46a2a-27fc-4d44-9d18-6d4553026a60] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port d7e16639-ee52-48ff-bbd6-ffe2b267f1d7, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 2eb46a2a-27fc-4d44-9d18-6d4553026a60 was re-scheduled: Binding failed for port d7e16639-ee52-48ff-bbd6-ffe2b267f1d7, please check neutron logs for more information.\n'] [ 561.790935] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 561.791323] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 2eb46a2a-27fc-4d44-9d18-6d4553026a60.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 2eb46a2a-27fc-4d44-9d18-6d4553026a60. [ 561.791864] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] [instance: 2eb46a2a-27fc-4d44-9d18-6d4553026a60] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 2eb46a2a-27fc-4d44-9d18-6d4553026a60. [ 561.823115] nova-conductor[54691]: DEBUG nova.network.neutron [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] [instance: 2eb46a2a-27fc-4d44-9d18-6d4553026a60] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 561.859360] nova-conductor[54691]: DEBUG nova.network.neutron [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] [instance: 2eb46a2a-27fc-4d44-9d18-6d4553026a60] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 561.868656] nova-conductor[54691]: DEBUG nova.network.neutron [None req-26cb9926-5a17-47ff-af0e-a897f86c2f17 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] [instance: 2eb46a2a-27fc-4d44-9d18-6d4553026a60] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 563.609175] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] [instance: 0b211be4-1955-456d-bf30-5195b5169807] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 3051a2c4-e501-45c9-b625-6fcae110ce27, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 0b211be4-1955-456d-bf30-5195b5169807 was re-scheduled: Binding failed for port 3051a2c4-e501-45c9-b625-6fcae110ce27, please check neutron logs for more information.\n'] [ 563.609530] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 563.609530] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 0b211be4-1955-456d-bf30-5195b5169807.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 0b211be4-1955-456d-bf30-5195b5169807. [ 563.609739] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] [instance: 0b211be4-1955-456d-bf30-5195b5169807] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 0b211be4-1955-456d-bf30-5195b5169807. [ 563.644626] nova-conductor[54691]: DEBUG nova.network.neutron [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] [instance: 0b211be4-1955-456d-bf30-5195b5169807] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 563.697023] nova-conductor[54691]: DEBUG nova.network.neutron [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] [instance: 0b211be4-1955-456d-bf30-5195b5169807] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 563.701619] nova-conductor[54691]: DEBUG nova.network.neutron [None req-e57d895d-64f7-4140-add7-4f167bcf3cf9 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447 tempest-FloatingIPsAssociationNegativeTestJSON-1846187447-project-member] [instance: 0b211be4-1955-456d-bf30-5195b5169807] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 563.833314] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Expecting reply to msg 9f63e17bcae9464f9130466ed92ec2e2 in queue reply_4316ed433d59463c9660e50df524570b [ 563.989436] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 9f63e17bcae9464f9130466ed92ec2e2 [ 564.338828] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 564.355648] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 564.355932] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 564.356136] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 564.388252] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 564.388495] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 564.388665] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 564.389085] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 564.389205] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 564.389392] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 564.409162] nova-conductor[54691]: DEBUG nova.quota [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Getting quotas for project 4346f2d2591d4970bf2f06626607e293. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 564.409162] nova-conductor[54691]: DEBUG nova.quota [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Getting quotas for user b899edcd821446b3ae6bc996b6a26037 and project 4346f2d2591d4970bf2f06626607e293. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 564.409162] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] [instance: a3182b4a-1846-4766-8988-aab0a74fa4c8] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 564.409437] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 564.410357] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 564.410583] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 564.415967] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] [instance: a3182b4a-1846-4766-8988-aab0a74fa4c8] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 564.416494] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 564.416700] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 564.416867] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 564.438870] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 564.438870] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 564.438870] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 565.207679] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Expecting reply to msg 8ae6a7053b9947ae8fdfa9d29dd6f30d in queue reply_4316ed433d59463c9660e50df524570b [ 565.350481] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 8ae6a7053b9947ae8fdfa9d29dd6f30d [ 565.710531] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 565.730023] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 565.730272] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 565.730446] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 565.773095] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 565.773349] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 565.773525] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 565.773901] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 565.774083] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 565.774235] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 565.788044] nova-conductor[54691]: DEBUG nova.quota [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Getting quotas for project 77a894bddd55484d956e0569e6d26d38. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 565.791566] nova-conductor[54691]: DEBUG nova.quota [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Getting quotas for user 87f04a89778e4350a90bcc998a96de0f and project 77a894bddd55484d956e0569e6d26d38. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 565.798653] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] [instance: c97b25ac-527f-404e-a539-1c0f43bcd66b] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 565.799142] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 565.799637] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 565.799637] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 565.812042] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] [instance: c97b25ac-527f-404e-a539-1c0f43bcd66b] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 565.812042] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 565.812042] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 565.812042] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 565.836543] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 565.836877] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 565.837134] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 566.343606] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Expecting reply to msg 3a816feba39941cbb571413d4264f612 in queue reply_4316ed433d59463c9660e50df524570b [ 566.499758] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 3a816feba39941cbb571413d4264f612 [ 566.846422] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 566.860096] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 566.860362] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 566.860547] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 566.894251] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 566.894492] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 566.894665] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 566.895029] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 566.895208] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 566.895364] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 566.917522] nova-conductor[54691]: DEBUG nova.quota [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Getting quotas for project 09e87409ac5e452896b8b112d0fd8722. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 566.921874] nova-conductor[54691]: DEBUG nova.quota [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Getting quotas for user c195c2561a8349baa3fbb8ab88c5d035 and project 09e87409ac5e452896b8b112d0fd8722. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 566.928894] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] [instance: b42fe7a4-ec30-441f-b5dc-1f705ee5c881] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 566.929631] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 566.930097] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 566.930471] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 566.938783] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] [instance: b42fe7a4-ec30-441f-b5dc-1f705ee5c881] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 566.939535] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 566.939742] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 566.939985] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 566.959103] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 566.959236] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 566.959445] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2431d4c3-8b63-4659-8b34-88717834f95b tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 568.996849] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Expecting reply to msg 6556f9e625bf4a48b0e01955bed294b0 in queue reply_4316ed433d59463c9660e50df524570b [ 569.150005] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Expecting reply to msg 8a6cec35b0ea42c8ba91ef1beb410750 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 569.217903] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 6556f9e625bf4a48b0e01955bed294b0 [ 569.338839] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 8a6cec35b0ea42c8ba91ef1beb410750 [ 569.504345] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 569.521057] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 569.521568] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 569.521874] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 569.572116] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 569.572354] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 569.572523] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 569.572947] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 569.573062] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 569.573219] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 569.582797] nova-conductor[54691]: DEBUG nova.quota [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Getting quotas for project ec2a7b4f1aea43cfa1972b508670852d. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 569.586478] nova-conductor[54691]: DEBUG nova.quota [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Getting quotas for user 811d5d3302b64a9ab3c07cbbcec100d4 and project ec2a7b4f1aea43cfa1972b508670852d. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 569.591913] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] [instance: 4e2f1fc0-102f-4d39-bec2-a2651918a202] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 569.592446] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 569.592647] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 569.592808] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 569.598587] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] [instance: 4e2f1fc0-102f-4d39-bec2-a2651918a202] block_device_mapping [BlockDeviceMapping(attachment_id=f4230336-3c40-4d38-b1a5-e44f521d0e02,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='volume',device_name=None,device_type=None,disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id=None,instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='volume',tag=None,updated_at=,uuid=,volume_id='e84143f7-236e-492d-9d69-0610b37fb002',volume_size=1,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 569.599308] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 569.599532] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 569.599698] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 569.618108] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 569.618345] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 569.618509] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 569.654133] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 569.671010] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 569.671010] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 569.671010] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 569.768146] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 569.768393] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 569.768845] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 569.768921] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 569.769112] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 569.769316] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 569.782871] nova-conductor[54692]: DEBUG nova.quota [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Getting quotas for project 09e87409ac5e452896b8b112d0fd8722. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 569.788461] nova-conductor[54692]: DEBUG nova.quota [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Getting quotas for user c195c2561a8349baa3fbb8ab88c5d035 and project 09e87409ac5e452896b8b112d0fd8722. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 569.794669] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] [instance: 13247e87-08ab-434e-b091-8b6116280fb7] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 569.795119] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 569.795323] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 569.795487] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 569.798148] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] [instance: 13247e87-08ab-434e-b091-8b6116280fb7] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 569.798814] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 569.799132] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 569.799324] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 569.824701] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 569.824937] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 569.825105] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-432886e9-c3a7-4fe4-bbf5-b0b28cdf5e39 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 572.171405] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=300,nova_cell1:UPDATE=130,nova_cell1:INSERT=23,nova_cell1:SAVEPOINT=24,nova_cell1:RELEASE=24 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 574.242855] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Expecting reply to msg 5c8cab44e6c742aabcd885db86a40c10 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 574.409101] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 5c8cab44e6c742aabcd885db86a40c10 [ 574.752215] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 574.779071] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 574.779385] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 574.780649] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 574.831761] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 574.832043] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 574.832333] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 574.832717] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 574.832950] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 574.833177] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 574.845792] nova-conductor[54692]: DEBUG nova.quota [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Getting quotas for project 14d7373c9b3445f3a36c0039ebd7e260. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 574.851773] nova-conductor[54692]: DEBUG nova.quota [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Getting quotas for user 213b05b97a734701ba942161e3b049e4 and project 14d7373c9b3445f3a36c0039ebd7e260. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 574.857908] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] [instance: ce17af27-45c3-461e-8cb7-b9b023d88361] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 574.858485] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 574.859124] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 574.859124] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 574.863837] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] [instance: ce17af27-45c3-461e-8cb7-b9b023d88361] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 574.864502] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 574.864710] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 574.864871] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 574.902839] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 574.902839] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 574.902839] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 575.682906] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] [instance: ae1041c6-888a-4cb3-8da3-00e43b5b6531] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 374282b5-a930-4685-a3e9-c381d14b89c8, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance ae1041c6-888a-4cb3-8da3-00e43b5b6531 was re-scheduled: Binding failed for port 374282b5-a930-4685-a3e9-c381d14b89c8, please check neutron logs for more information.\n'] [ 575.683619] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 575.684100] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ae1041c6-888a-4cb3-8da3-00e43b5b6531.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ae1041c6-888a-4cb3-8da3-00e43b5b6531. [ 575.684438] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] [instance: ae1041c6-888a-4cb3-8da3-00e43b5b6531] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ae1041c6-888a-4cb3-8da3-00e43b5b6531. [ 575.718043] nova-conductor[54692]: DEBUG nova.network.neutron [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] [instance: ae1041c6-888a-4cb3-8da3-00e43b5b6531] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 575.743910] nova-conductor[54692]: DEBUG nova.network.neutron [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] [instance: ae1041c6-888a-4cb3-8da3-00e43b5b6531] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 575.745410] nova-conductor[54692]: DEBUG nova.network.neutron [None req-d95615ff-60d5-405e-97b1-3e2c394f2151 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] [instance: ae1041c6-888a-4cb3-8da3-00e43b5b6531] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 576.530704] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:SELECT=324,nova_cell1:UPDATE=140,nova_cell1:INSERT=18,nova_cell1:SAVEPOINT=27,nova_cell1:RELEASE=27 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 577.383056] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Expecting reply to msg 633540e3d7eb4cfeb42168422c363d60 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 577.573333] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 633540e3d7eb4cfeb42168422c363d60 [ 577.886402] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 577.889434] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: aa9671f3-07a8-422f-93b5-006de0f2cb3c] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port ea932e45-f256-494c-9fc5-5ace73092f2f, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance aa9671f3-07a8-422f-93b5-006de0f2cb3c was re-scheduled: Binding failed for port ea932e45-f256-494c-9fc5-5ace73092f2f, please check neutron logs for more information.\n'] [ 577.890104] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 577.890352] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance aa9671f3-07a8-422f-93b5-006de0f2cb3c.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance aa9671f3-07a8-422f-93b5-006de0f2cb3c. [ 577.890683] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: aa9671f3-07a8-422f-93b5-006de0f2cb3c] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance aa9671f3-07a8-422f-93b5-006de0f2cb3c. [ 577.914694] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 577.914694] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 577.918136] nova-conductor[54691]: DEBUG nova.network.neutron [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: aa9671f3-07a8-422f-93b5-006de0f2cb3c] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 577.922297] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 577.941391] nova-conductor[54691]: DEBUG nova.network.neutron [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: aa9671f3-07a8-422f-93b5-006de0f2cb3c] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 577.946419] nova-conductor[54691]: DEBUG nova.network.neutron [None req-a5c29c37-c3c6-492e-b681-b76ce126536f tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: aa9671f3-07a8-422f-93b5-006de0f2cb3c] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 577.962302] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 577.962593] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 577.962770] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 577.963138] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 577.963325] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 577.963482] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 577.972434] nova-conductor[54692]: DEBUG nova.quota [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Getting quotas for project 2848f00a047a4e638b81a83e20c4c182. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 577.980256] nova-conductor[54692]: DEBUG nova.quota [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Getting quotas for user 252ad3448fca4b76b3dce258c6f00107 and project 2848f00a047a4e638b81a83e20c4c182. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 577.989072] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] [instance: 9a445d5c-7a5f-4149-b198-306a04b4c040] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 577.989620] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 577.989842] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 577.990009] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 577.993258] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] [instance: 9a445d5c-7a5f-4149-b198-306a04b4c040] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 577.993913] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 577.994118] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 577.994277] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 578.014179] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 578.014402] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 578.014567] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 578.286242] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Expecting reply to msg e2656c45e33a417c99de8804ba97a437 in queue reply_4316ed433d59463c9660e50df524570b [ 578.410898] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg e2656c45e33a417c99de8804ba97a437 [ 578.795525] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 578.814822] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 578.815051] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 578.815215] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 578.857026] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 578.857254] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 578.857449] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 578.857822] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 578.858001] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 578.858155] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 578.868301] nova-conductor[54691]: DEBUG nova.quota [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Getting quotas for project d781d0a045444391bc070520be1b6431. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 578.872272] nova-conductor[54691]: DEBUG nova.quota [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Getting quotas for user 6a6d2a917fec48e48e943137dff043fd and project d781d0a045444391bc070520be1b6431. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 578.884130] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] [instance: ad008294-52db-40de-95ad-3e0f2d30a462] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 578.884609] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 578.884828] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 578.885001] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 578.889691] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] [instance: ad008294-52db-40de-95ad-3e0f2d30a462] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 578.890292] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 578.890749] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 578.890928] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 578.912360] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 578.912360] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 578.912360] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 580.128531] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Expecting reply to msg e689c3986933405fbd5a1ce543c3b3b5 in queue reply_4316ed433d59463c9660e50df524570b [ 580.316672] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg e689c3986933405fbd5a1ce543c3b3b5 [ 580.618698] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Expecting reply to msg bbe77703a2ec45029890d1c519ce3fa6 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 580.635076] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 580.665003] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 580.665396] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 580.665619] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 580.743753] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 580.743872] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 580.744077] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 580.744498] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 580.744695] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 580.744925] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 580.781158] nova-conductor[54691]: DEBUG nova.quota [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Getting quotas for project 2b6ce1f734f74a948ee720d8b43795b0. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 580.793066] nova-conductor[54691]: DEBUG nova.quota [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Getting quotas for user cf477a6798c9449bbb615277de8c1170 and project 2b6ce1f734f74a948ee720d8b43795b0. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 580.804738] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] [instance: 38767b46-9a31-4031-981b-424497dc4a34] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 580.805385] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 580.805524] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 580.805683] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 580.817860] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg bbe77703a2ec45029890d1c519ce3fa6 [ 580.823996] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] [instance: 38767b46-9a31-4031-981b-424497dc4a34] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 580.828025] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 580.828436] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 580.829510] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 580.870298] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 580.870628] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 580.870916] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 581.120932] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 581.138409] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 581.138648] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 581.138983] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 581.187051] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 581.187611] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 581.187965] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 581.188727] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 581.189063] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 581.189428] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 581.199218] nova-conductor[54692]: DEBUG nova.quota [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Getting quotas for project de518d899df341b9949e44aa38f28a39. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 581.218428] nova-conductor[54692]: DEBUG nova.quota [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Getting quotas for user 433c8222844b4131b191877d227b9e8c and project de518d899df341b9949e44aa38f28a39. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 581.228059] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] [instance: 6480b28b-7ded-46c1-8fa0-6e1abba4bf7a] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 581.228781] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 581.229152] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 581.229490] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 581.237688] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] [instance: 6480b28b-7ded-46c1-8fa0-6e1abba4bf7a] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 581.239089] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 581.239490] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 581.239828] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 581.265884] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 581.266387] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 581.266976] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 581.310545] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] [instance: a5a690d3-8018-4cc5-8d59-55062a595ed0] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 42afc0e2-7da7-431c-a861-5c77b0a53036, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance a5a690d3-8018-4cc5-8d59-55062a595ed0 was re-scheduled: Binding failed for port 42afc0e2-7da7-431c-a861-5c77b0a53036, please check neutron logs for more information.\n'] [ 581.311196] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 581.311481] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance a5a690d3-8018-4cc5-8d59-55062a595ed0.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance a5a690d3-8018-4cc5-8d59-55062a595ed0. [ 581.311768] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] [instance: a5a690d3-8018-4cc5-8d59-55062a595ed0] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance a5a690d3-8018-4cc5-8d59-55062a595ed0. [ 581.369461] nova-conductor[54691]: DEBUG nova.network.neutron [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] [instance: a5a690d3-8018-4cc5-8d59-55062a595ed0] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 581.400948] nova-conductor[54691]: DEBUG nova.network.neutron [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] [instance: a5a690d3-8018-4cc5-8d59-55062a595ed0] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 581.418717] nova-conductor[54691]: DEBUG nova.network.neutron [None req-552db156-4d36-4ab3-b539-a9f13c2fd531 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] [instance: a5a690d3-8018-4cc5-8d59-55062a595ed0] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 581.696116] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Expecting reply to msg 43c3c489e1914c0b8c99aee7b1c2bb54 in queue reply_4316ed433d59463c9660e50df524570b [ 581.925005] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 43c3c489e1914c0b8c99aee7b1c2bb54 [ 582.204355] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 582.230141] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 582.230141] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 582.230141] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 582.276269] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 582.276623] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 582.276902] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 582.277876] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 582.278238] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 582.278458] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 582.297231] nova-conductor[54691]: DEBUG nova.quota [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Getting quotas for project 532d5738736643d2836c0896012494da. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 582.300720] nova-conductor[54691]: DEBUG nova.quota [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Getting quotas for user 16306c5f0e4d4eabbcce4a9c72edce5f and project 532d5738736643d2836c0896012494da. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 582.306800] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] [instance: 41650a98-fda7-44fd-aca6-d3b94cf17308] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 582.307288] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 582.307512] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 582.307678] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 582.313974] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] [instance: 41650a98-fda7-44fd-aca6-d3b94cf17308] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 582.313974] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 582.313974] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 582.313974] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 582.329136] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 582.329465] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 582.329719] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 583.078408] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Expecting reply to msg e6b04f89dc91493bbf38be640f4c214d in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 583.268402] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg e6b04f89dc91493bbf38be640f4c214d [ 583.575644] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 583.589228] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 583.589401] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 583.589518] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 583.616885] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 583.616992] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 583.617162] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 583.617715] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 583.617816] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 583.617917] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 583.626583] nova-conductor[54692]: DEBUG nova.quota [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Getting quotas for project 4dd6c351f2f24b669fcd5add934e1546. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 583.628783] nova-conductor[54692]: DEBUG nova.quota [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Getting quotas for user 1745c807cc6b4817acff209027fabc8c and project 4dd6c351f2f24b669fcd5add934e1546. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 583.635046] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] [instance: 3e82fa5d-6610-4d3d-972f-4120aca5e8a7] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 583.635046] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 583.635046] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 583.635046] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 583.639637] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] [instance: 3e82fa5d-6610-4d3d-972f-4120aca5e8a7] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 583.640368] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 583.640613] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 583.640790] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 583.663003] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 583.663257] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 583.663494] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 584.116693] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Expecting reply to msg 80ffbc7cc3094a218301414df638ff0d in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 584.274905] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 80ffbc7cc3094a218301414df638ff0d [ 584.621598] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 584.634837] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 584.643165] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 584.643416] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.008s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 584.686223] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 584.686799] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 584.687126] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 584.687786] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 584.688516] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 584.688809] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 584.712923] nova-conductor[54692]: DEBUG nova.quota [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Getting quotas for project c43001da6eb843efbcbee6303e4a6d00. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 584.712923] nova-conductor[54692]: DEBUG nova.quota [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Getting quotas for user b5619794672b4d01a88e58fb3c17e65c and project c43001da6eb843efbcbee6303e4a6d00. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 584.722214] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] [instance: 15215512-da0c-407f-a352-43c051698f2d] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 584.723376] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 584.723718] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 584.724029] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 584.729687] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] [instance: 15215512-da0c-407f-a352-43c051698f2d] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 584.730697] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 584.730823] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 584.731026] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 584.751631] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 584.751860] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 584.752102] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 584.771098] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Expecting reply to msg 57adbe15f2fd4b88919e6e2f135ce942 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 584.923446] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 57adbe15f2fd4b88919e6e2f135ce942 [ 585.130164] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Expecting reply to msg fbd2876e6c784ffb8861ffe9d95f070f in queue reply_4316ed433d59463c9660e50df524570b [ 585.269798] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg fbd2876e6c784ffb8861ffe9d95f070f [ 585.274960] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 585.288153] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 585.288545] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 585.289016] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 585.315282] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 585.315666] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 585.316167] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 585.316994] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 585.317405] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 585.317684] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 585.327791] nova-conductor[54692]: DEBUG nova.quota [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Getting quotas for project 204d8f668a894827994c416d74f34c18. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 585.330897] nova-conductor[54692]: DEBUG nova.quota [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Getting quotas for user ea7e09048589421e96a4d19f8e884c8a and project 204d8f668a894827994c416d74f34c18. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 585.336269] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: c4512101-c88c-4f3c-bde1-def0f3520101] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 585.336856] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 585.337173] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 585.337463] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 585.341412] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: c4512101-c88c-4f3c-bde1-def0f3520101] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 585.342222] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 585.342544] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 585.342824] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 585.359895] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 585.360144] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 585.360337] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 585.632802] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 585.637323] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=100,nova_api:UPDATE=9,nova_api:DELETE=7 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 585.645441] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 585.645734] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 585.645950] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 585.687658] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 585.687974] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 585.688236] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 585.688710] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 585.688925] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 585.689310] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 585.701029] nova-conductor[54691]: DEBUG nova.quota [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Getting quotas for project 3bca108d42ad437da93a84c160dcade4. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 585.704208] nova-conductor[54691]: DEBUG nova.quota [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Getting quotas for user b6684e39ea434b0cbd9623d38b1e68e0 and project 3bca108d42ad437da93a84c160dcade4. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 585.709464] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] [instance: b88d76a9-6a30-4d5f-a5cd-f698e005dab5] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 585.709999] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 585.710386] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 585.710571] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 585.717658] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] [instance: b88d76a9-6a30-4d5f-a5cd-f698e005dab5] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 585.718301] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 585.718557] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 585.718798] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 585.751874] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 585.752198] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 585.752436] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 585.988506] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Expecting reply to msg b2172ca83cf94609977088a7e0e2c5ee in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 586.132750] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg b2172ca83cf94609977088a7e0e2c5ee [ 586.490697] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 586.503176] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 586.503394] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 586.503557] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 586.535695] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 586.535932] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 586.536124] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 586.536846] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 586.537192] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 586.537403] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 586.552168] nova-conductor[54692]: DEBUG nova.quota [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Getting quotas for project 00550bdf4bcd46fab1979767622472b5. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 586.555985] nova-conductor[54692]: DEBUG nova.quota [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Getting quotas for user 3363a3996f484f75ad17c48432c970e7 and project 00550bdf4bcd46fab1979767622472b5. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 586.563468] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] [instance: eeb7cdc7-427f-496f-be5d-60151a4bb0f3] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 586.563957] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 586.564230] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 586.564406] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 586.572703] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] [instance: eeb7cdc7-427f-496f-be5d-60151a4bb0f3] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 586.573412] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 586.573652] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 586.573830] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 586.607048] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 586.607487] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 586.607822] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 587.173823] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Expecting reply to msg 65ebdbd19d064b38b1dc449961bdde19 in queue reply_4316ed433d59463c9660e50df524570b [ 587.326359] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 65ebdbd19d064b38b1dc449961bdde19 [ 587.676543] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 587.685161] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=112,nova_api:DELETE=11,nova_api:UPDATE=9 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 587.702953] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 587.703192] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 587.703372] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 587.705573] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:SELECT=56,nova_cell1:SAVEPOINT=15,nova_cell1:INSERT=142,nova_cell1:RELEASE=15 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 587.742117] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 587.742372] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 587.742539] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 587.743283] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 587.743676] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 587.744122] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 587.749368] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell0:SELECT=20 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 587.758586] nova-conductor[54691]: DEBUG nova.quota [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Getting quotas for project 00550bdf4bcd46fab1979767622472b5. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 587.762638] nova-conductor[54691]: DEBUG nova.quota [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Getting quotas for user 3363a3996f484f75ad17c48432c970e7 and project 00550bdf4bcd46fab1979767622472b5. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 587.769206] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] [instance: 1754ee22-ae95-4439-a6c0-200fd153bad0] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 587.770001] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 587.770215] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 587.770416] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 587.775440] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] [instance: 1754ee22-ae95-4439-a6c0-200fd153bad0] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 587.776092] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 587.776476] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 587.776656] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 587.796915] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 587.797191] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 587.798603] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 588.633073] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Expecting reply to msg 14f27d5a7ff94cfd86a66d69cb63b5b4 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 588.840685] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 14f27d5a7ff94cfd86a66d69cb63b5b4 [ 589.141911] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 589.154650] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 589.154880] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 589.155050] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 589.188095] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 589.188095] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 589.188095] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 589.188095] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 589.188095] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 589.188095] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 589.195788] nova-conductor[54692]: DEBUG nova.quota [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Getting quotas for project 204d8f668a894827994c416d74f34c18. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 589.199240] nova-conductor[54692]: DEBUG nova.quota [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Getting quotas for user ea7e09048589421e96a4d19f8e884c8a and project 204d8f668a894827994c416d74f34c18. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 589.204458] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: 71ce0048-90fb-460c-a099-7eff5c092d35] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 589.204951] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 589.205156] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 589.205322] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 589.209745] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: 71ce0048-90fb-460c-a099-7eff5c092d35] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 589.213160] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 589.213160] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 589.213160] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 589.227320] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 589.227320] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 589.227491] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 590.754915] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Expecting reply to msg 813a6cde40084f039be439602fcf3b8b in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 590.882591] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 813a6cde40084f039be439602fcf3b8b [ 591.260572] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 591.273648] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 591.273863] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 591.274028] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 591.297432] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 591.297842] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 591.297842] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 591.298352] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 591.298556] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 591.298657] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 591.307253] nova-conductor[54692]: DEBUG nova.quota [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Getting quotas for project 7f30e3b24f414007b0066cab5346a3bf. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 591.310524] nova-conductor[54692]: DEBUG nova.quota [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Getting quotas for user 71886e19fbc749d989c55e17c178862a and project 7f30e3b24f414007b0066cab5346a3bf. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 591.325236] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] [instance: 20d9b891-9c2a-479c-8fd8-93405480dee4] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 591.325749] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 591.325948] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 591.326109] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 591.333596] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] [instance: 20d9b891-9c2a-479c-8fd8-93405480dee4] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 591.334265] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 591.334465] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 591.334627] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 591.355570] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 591.355788] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 591.355953] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 592.469537] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Expecting reply to msg 448b3f6626214c44b7dc6bd747b06482 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 592.619201] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 448b3f6626214c44b7dc6bd747b06482 [ 592.970054] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 592.990795] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 592.991106] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.002s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 592.991302] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 593.033655] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 593.033655] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 593.033655] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 593.034080] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 593.034080] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 593.034253] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 593.059153] nova-conductor[54692]: DEBUG nova.quota [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Getting quotas for project 3eb283a2654947978d60cee9a47d568f. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 593.064053] nova-conductor[54692]: DEBUG nova.quota [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Getting quotas for user 520a746e985543d887fe9436704586f2 and project 3eb283a2654947978d60cee9a47d568f. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 593.078418] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] [instance: 48a0d4f8-4fab-454b-b065-e552647fba3f] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 593.079208] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 593.079475] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 593.079663] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 593.094657] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] [instance: 48a0d4f8-4fab-454b-b065-e552647fba3f] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 593.100624] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 593.100973] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.006s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 593.101211] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 593.128876] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 593.129117] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 593.129328] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 597.779616] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:INSERT=7,nova_cell1:SELECT=2 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 597.790170] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=12 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 597.793411] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=3,nova_api:UPDATE=1,nova_api:DELETE=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 598.379733] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] [instance: 01b348db-6a0d-4283-b9d9-8edacfd1c219] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port d3ab0149-beaf-4cc6-a12c-69e4bba83062, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 01b348db-6a0d-4283-b9d9-8edacfd1c219 was re-scheduled: Binding failed for port d3ab0149-beaf-4cc6-a12c-69e4bba83062, please check neutron logs for more information.\n'] [ 598.380754] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 598.380754] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 01b348db-6a0d-4283-b9d9-8edacfd1c219.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 01b348db-6a0d-4283-b9d9-8edacfd1c219. [ 598.381163] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] [instance: 01b348db-6a0d-4283-b9d9-8edacfd1c219] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 01b348db-6a0d-4283-b9d9-8edacfd1c219. [ 598.401975] nova-conductor[54692]: DEBUG nova.network.neutron [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] [instance: 01b348db-6a0d-4283-b9d9-8edacfd1c219] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 598.435395] nova-conductor[54692]: DEBUG nova.network.neutron [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] [instance: 01b348db-6a0d-4283-b9d9-8edacfd1c219] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 598.439461] nova-conductor[54692]: DEBUG nova.network.neutron [None req-aa227588-b68c-441d-90f4-890883f45639 tempest-ListServerFiltersTestJSON-22237429 tempest-ListServerFiltersTestJSON-22237429-project-member] [instance: 01b348db-6a0d-4283-b9d9-8edacfd1c219] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 600.781425] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Expecting reply to msg 9add1c88c6224171b5acd5b2991b0677 in queue reply_4316ed433d59463c9660e50df524570b [ 601.041579] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 9add1c88c6224171b5acd5b2991b0677 [ 601.236552] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Expecting reply to msg 7999dea80ee9498d95c2f79cf0ec70b5 in queue reply_4316ed433d59463c9660e50df524570b [ 601.285610] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Took 0.50 seconds to select destinations for 3 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 601.310388] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.310613] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.310784] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.367370] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.367610] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.367783] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.420356] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 7999dea80ee9498d95c2f79cf0ec70b5 [ 601.424049] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.424288] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.424457] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.465318] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.465570] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.465741] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.466119] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.466322] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.466489] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.478802] nova-conductor[54691]: DEBUG nova.quota [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Getting quotas for project 78c69a6dc9124f71b9a1f84fb2da5d4d. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 601.482269] nova-conductor[54691]: DEBUG nova.quota [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Getting quotas for user c2b0969731654200b597f9d6398c2cfb and project 78c69a6dc9124f71b9a1f84fb2da5d4d. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 601.489560] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] [instance: 459055aa-ae56-480f-b2c6-e45cc05bb50c] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 601.490082] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.490317] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.490511] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.493627] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] [instance: 459055aa-ae56-480f-b2c6-e45cc05bb50c] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 601.494329] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.494544] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.494716] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.519602] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.519746] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.519914] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.528064] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] [instance: 5041b2c1-e7f1-4039-b7c4-1f677918025a] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 601.529830] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.529830] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.529830] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.537847] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] [instance: 5041b2c1-e7f1-4039-b7c4-1f677918025a] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 601.537847] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.537847] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.537847] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.585699] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.585699] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.585699] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.597511] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] [instance: 896c2790-9107-4d1c-811b-07a87582c0a1] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 601.598184] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.598267] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.598407] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.603111] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] [instance: 896c2790-9107-4d1c-811b-07a87582c0a1] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 601.603787] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.603987] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.604183] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.621709] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.621967] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.622141] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.738767] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 601.779945] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.782108] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.782108] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.850839] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.851103] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.851287] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.851707] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.851893] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.852085] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.862973] nova-conductor[54691]: DEBUG nova.quota [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Getting quotas for project f160fdbb9ae34ba3a4a17c903121bc49. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 601.867235] nova-conductor[54691]: DEBUG nova.quota [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Getting quotas for user 717a6eb5643b4f188ec28201b885675e and project f160fdbb9ae34ba3a4a17c903121bc49. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 601.874703] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] [instance: 1174dcea-6ba6-4a2b-9eb4-85e662691c30] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 601.875206] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.875410] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.875927] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.887656] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] [instance: 1174dcea-6ba6-4a2b-9eb4-85e662691c30] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 601.888443] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.888717] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.888909] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.916520] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.917231] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.917795] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 603.038202] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell0:SELECT=11 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 603.103627] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=33,nova_cell1:INSERT=84,nova_cell1:SAVEPOINT=7,nova_cell1:RELEASE=7 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 603.118292] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=67,nova_api:UPDATE=4,nova_api:DELETE=4 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 603.124171] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=65,nova_api:DELETE=7,nova_api:UPDATE=7 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 606.597743] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Expecting reply to msg ac86a8100e5042759455ce93a216f263 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 606.733683] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg ac86a8100e5042759455ce93a216f263 [ 607.100077] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 607.116084] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 607.116379] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 607.120138] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 607.192909] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 607.193649] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 607.194287] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 607.195002] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 607.195475] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 607.195782] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 607.220102] nova-conductor[54692]: DEBUG nova.quota [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Getting quotas for project 490a56a64a044ddf955e8c0056890140. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 607.233961] nova-conductor[54692]: DEBUG nova.quota [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Getting quotas for user e5776ada81cb4526966ec39aad15fc4f and project 490a56a64a044ddf955e8c0056890140. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 607.239671] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] [instance: 1a5a4c2e-74e6-4a62-adda-c3d89deecf05] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 607.240474] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 607.240806] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 607.241114] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 607.249794] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] [instance: 1a5a4c2e-74e6-4a62-adda-c3d89deecf05] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 607.250790] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 607.251131] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 607.251481] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 607.700955] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 607.701217] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 607.702349] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 608.141557] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] [instance: 958e85c0-cb75-4dfd-92c4-b68957e87ec5] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 3fe70ba4-a5cc-4008-a5e7-05d2f9a3ea50, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 958e85c0-cb75-4dfd-92c4-b68957e87ec5 was re-scheduled: Binding failed for port 3fe70ba4-a5cc-4008-a5e7-05d2f9a3ea50, please check neutron logs for more information.\n'] [ 608.142267] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 608.142502] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 958e85c0-cb75-4dfd-92c4-b68957e87ec5.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 958e85c0-cb75-4dfd-92c4-b68957e87ec5. [ 608.142811] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] [instance: 958e85c0-cb75-4dfd-92c4-b68957e87ec5] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 958e85c0-cb75-4dfd-92c4-b68957e87ec5. [ 608.170092] nova-conductor[54692]: DEBUG nova.network.neutron [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] [instance: 958e85c0-cb75-4dfd-92c4-b68957e87ec5] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 608.185236] nova-conductor[54692]: DEBUG nova.network.neutron [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] [instance: 958e85c0-cb75-4dfd-92c4-b68957e87ec5] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 608.188164] nova-conductor[54692]: DEBUG nova.network.neutron [None req-662d052f-b537-404d-ba0d-b15485fa3d1f tempest-ImagesNegativeTestJSON-1008518018 tempest-ImagesNegativeTestJSON-1008518018-project-member] [instance: 958e85c0-cb75-4dfd-92c4-b68957e87ec5] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 610.321833] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] [instance: 52f8e827-3faf-47c1-b793-ccacc5a0bc4c] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port e6cc30e4-2174-44e6-a3ca-ec741165ee60, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 52f8e827-3faf-47c1-b793-ccacc5a0bc4c was re-scheduled: Binding failed for port e6cc30e4-2174-44e6-a3ca-ec741165ee60, please check neutron logs for more information.\n'] [ 610.322414] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 610.322650] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 52f8e827-3faf-47c1-b793-ccacc5a0bc4c.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 52f8e827-3faf-47c1-b793-ccacc5a0bc4c. [ 610.322856] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] [instance: 52f8e827-3faf-47c1-b793-ccacc5a0bc4c] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 52f8e827-3faf-47c1-b793-ccacc5a0bc4c. [ 610.359399] nova-conductor[54692]: DEBUG nova.network.neutron [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] [instance: 52f8e827-3faf-47c1-b793-ccacc5a0bc4c] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 610.384111] nova-conductor[54692]: DEBUG nova.network.neutron [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] [instance: 52f8e827-3faf-47c1-b793-ccacc5a0bc4c] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 610.387254] nova-conductor[54692]: DEBUG nova.network.neutron [None req-fcd30cc0-1946-4ab4-a017-ab01550a2c99 tempest-ServerExternalEventsTest-1600310199 tempest-ServerExternalEventsTest-1600310199-project-member] [instance: 52f8e827-3faf-47c1-b793-ccacc5a0bc4c] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 611.855423] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell0:SELECT=2 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 611.891594] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:SELECT=10,nova_cell1:SAVEPOINT=2,nova_cell1:INSERT=30,nova_cell1:RELEASE=2 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 611.901705] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=23,nova_api:DELETE=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 611.906712] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=9,nova_api:UPDATE=4,nova_api:DELETE=3 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 612.380862] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] [instance: f537a3a9-b63a-4233-968a-942aca6ada0e] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 3b1dae23-7093-47dd-9cbe-fa5054998b51, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance f537a3a9-b63a-4233-968a-942aca6ada0e was re-scheduled: Binding failed for port 3b1dae23-7093-47dd-9cbe-fa5054998b51, please check neutron logs for more information.\n'] [ 612.381619] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 612.381785] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f537a3a9-b63a-4233-968a-942aca6ada0e.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f537a3a9-b63a-4233-968a-942aca6ada0e. [ 612.382422] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] [instance: f537a3a9-b63a-4233-968a-942aca6ada0e] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f537a3a9-b63a-4233-968a-942aca6ada0e. [ 612.406348] nova-conductor[54691]: DEBUG nova.network.neutron [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] [instance: f537a3a9-b63a-4233-968a-942aca6ada0e] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 612.427606] nova-conductor[54691]: DEBUG nova.network.neutron [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] [instance: f537a3a9-b63a-4233-968a-942aca6ada0e] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 612.430681] nova-conductor[54691]: DEBUG nova.network.neutron [None req-098d58f7-87c6-4589-b09b-42ae718b38e8 tempest-TenantUsagesTestJSON-2051145345 tempest-TenantUsagesTestJSON-2051145345-project-member] [instance: f537a3a9-b63a-4233-968a-942aca6ada0e] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 614.391372] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] [instance: bd41759b-5c32-42e6-b625-4ea48f98045e] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 74c43113-48d4-426b-9dd2-405b9e6ab39f, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance bd41759b-5c32-42e6-b625-4ea48f98045e was re-scheduled: Binding failed for port 74c43113-48d4-426b-9dd2-405b9e6ab39f, please check neutron logs for more information.\n'] [ 614.391372] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 614.391372] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance bd41759b-5c32-42e6-b625-4ea48f98045e.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance bd41759b-5c32-42e6-b625-4ea48f98045e. [ 614.391372] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] [instance: bd41759b-5c32-42e6-b625-4ea48f98045e] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance bd41759b-5c32-42e6-b625-4ea48f98045e. [ 614.421929] nova-conductor[54691]: DEBUG nova.network.neutron [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] [instance: bd41759b-5c32-42e6-b625-4ea48f98045e] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 614.444352] nova-conductor[54691]: DEBUG nova.network.neutron [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] [instance: bd41759b-5c32-42e6-b625-4ea48f98045e] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 614.449390] nova-conductor[54691]: DEBUG nova.network.neutron [None req-2cbeab54-ce31-4fd2-a93e-cef8b9a60b8c tempest-VolumesAssistedSnapshotsTest-161195681 tempest-VolumesAssistedSnapshotsTest-161195681-project-member] [instance: bd41759b-5c32-42e6-b625-4ea48f98045e] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 616.463416] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] [instance: 25508ff0-bc62-47b3-ae9e-7d9d8c735edc] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port db4fac68-33e9-43e4-a979-d71a08eb0e06, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 25508ff0-bc62-47b3-ae9e-7d9d8c735edc was re-scheduled: Binding failed for port db4fac68-33e9-43e4-a979-d71a08eb0e06, please check neutron logs for more information.\n'] [ 616.465053] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 616.465273] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 25508ff0-bc62-47b3-ae9e-7d9d8c735edc.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 25508ff0-bc62-47b3-ae9e-7d9d8c735edc. [ 616.465507] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] [instance: 25508ff0-bc62-47b3-ae9e-7d9d8c735edc] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 25508ff0-bc62-47b3-ae9e-7d9d8c735edc. [ 616.492684] nova-conductor[54692]: DEBUG nova.network.neutron [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] [instance: 25508ff0-bc62-47b3-ae9e-7d9d8c735edc] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 616.528379] nova-conductor[54692]: DEBUG nova.network.neutron [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] [instance: 25508ff0-bc62-47b3-ae9e-7d9d8c735edc] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 616.532840] nova-conductor[54692]: DEBUG nova.network.neutron [None req-b05dba1a-f8ea-4623-ac42-9defb01479a1 tempest-InstanceActionsTestJSON-1210756679 tempest-InstanceActionsTestJSON-1210756679-project-member] [instance: 25508ff0-bc62-47b3-ae9e-7d9d8c735edc] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 617.202160] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell0:SELECT=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 617.633483] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=3,nova_cell1:SAVEPOINT=1,nova_cell1:INSERT=8,nova_cell1:RELEASE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 617.674992] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=7 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 617.694059] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=5,nova_api:UPDATE=1,nova_api:DELETE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 618.700943] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] [instance: 918aa256-a10d-46d1-b9a4-ecd70cf3566f] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port d1d9c4d7-2cd5-43d7-8471-60c524b7cfda, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 918aa256-a10d-46d1-b9a4-ecd70cf3566f was re-scheduled: Binding failed for port d1d9c4d7-2cd5-43d7-8471-60c524b7cfda, please check neutron logs for more information.\n'] [ 618.705053] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 618.705053] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 918aa256-a10d-46d1-b9a4-ecd70cf3566f.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 918aa256-a10d-46d1-b9a4-ecd70cf3566f. [ 618.705053] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] [instance: 918aa256-a10d-46d1-b9a4-ecd70cf3566f] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 918aa256-a10d-46d1-b9a4-ecd70cf3566f. [ 618.733467] nova-conductor[54691]: DEBUG nova.network.neutron [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] [instance: 918aa256-a10d-46d1-b9a4-ecd70cf3566f] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 618.758904] nova-conductor[54691]: DEBUG nova.network.neutron [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] [instance: 918aa256-a10d-46d1-b9a4-ecd70cf3566f] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 618.762795] nova-conductor[54691]: DEBUG nova.network.neutron [None req-f3e62278-36b5-4c99-9cae-928a9ac51e8f tempest-ServerActionsTestOtherB-819696560 tempest-ServerActionsTestOtherB-819696560-project-member] [instance: 918aa256-a10d-46d1-b9a4-ecd70cf3566f] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 622.870257] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Expecting reply to msg 3d076c993e49439fa68b1d6db327d224 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 623.060443] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 3d076c993e49439fa68b1d6db327d224 [ 623.373143] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 623.383087] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] [instance: 64ee370c-6d43-4e2b-a27a-3aa716fdc322] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 2fc1b9dc-5720-4a4e-8ecd-4e25c684be47, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 64ee370c-6d43-4e2b-a27a-3aa716fdc322 was re-scheduled: Binding failed for port 2fc1b9dc-5720-4a4e-8ecd-4e25c684be47, please check neutron logs for more information.\n'] [ 623.383643] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 623.383872] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 64ee370c-6d43-4e2b-a27a-3aa716fdc322.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 64ee370c-6d43-4e2b-a27a-3aa716fdc322. [ 623.384291] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] [instance: 64ee370c-6d43-4e2b-a27a-3aa716fdc322] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 64ee370c-6d43-4e2b-a27a-3aa716fdc322. [ 623.405032] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 623.405246] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 623.405665] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 623.433080] nova-conductor[54692]: DEBUG nova.network.neutron [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] [instance: 64ee370c-6d43-4e2b-a27a-3aa716fdc322] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 623.447732] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 623.447960] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 623.448157] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 623.448523] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 623.448710] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 623.448870] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 623.460061] nova-conductor[54692]: DEBUG nova.quota [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Getting quotas for project dab569c547cb41cea49d87164ed92136. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 623.463487] nova-conductor[54692]: DEBUG nova.quota [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Getting quotas for user 43c10ef2ec3c40ea985d9d1213f25e6b and project dab569c547cb41cea49d87164ed92136. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 623.471139] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] [instance: 427400bc-f265-40cb-8c1c-ccab7bc94b1c] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 623.471568] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 623.471796] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 623.471962] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 623.474130] nova-conductor[54692]: DEBUG nova.network.neutron [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] [instance: 64ee370c-6d43-4e2b-a27a-3aa716fdc322] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 623.477009] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] [instance: 427400bc-f265-40cb-8c1c-ccab7bc94b1c] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 623.477941] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 623.477941] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 623.478046] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 623.480690] nova-conductor[54692]: DEBUG nova.network.neutron [None req-4d8ab2ec-6616-469b-bdfc-cefe695f9c27 tempest-ServersWithSpecificFlavorTestJSON-1147380317 tempest-ServersWithSpecificFlavorTestJSON-1147380317-project-member] [instance: 64ee370c-6d43-4e2b-a27a-3aa716fdc322] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 623.509567] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 623.509793] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 623.509962] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 625.268730] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] [instance: 3a848010-382b-4753-8420-37222963e7e2] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port ad867a88-c3d1-405e-be2f-5e3328b9c9b9, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 3a848010-382b-4753-8420-37222963e7e2 was re-scheduled: Binding failed for port ad867a88-c3d1-405e-be2f-5e3328b9c9b9, please check neutron logs for more information.\n'] [ 625.269442] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 625.269760] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 3a848010-382b-4753-8420-37222963e7e2.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 3a848010-382b-4753-8420-37222963e7e2. [ 625.271375] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] [instance: 3a848010-382b-4753-8420-37222963e7e2] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 3a848010-382b-4753-8420-37222963e7e2. [ 625.305882] nova-conductor[54692]: DEBUG nova.network.neutron [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] [instance: 3a848010-382b-4753-8420-37222963e7e2] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 625.339508] nova-conductor[54692]: DEBUG nova.network.neutron [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] [instance: 3a848010-382b-4753-8420-37222963e7e2] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 625.347564] nova-conductor[54692]: DEBUG nova.network.neutron [None req-c171efca-4e75-4a68-8b69-3d25d17b5305 tempest-FloatingIPsAssociationTestJSON-1881166845 tempest-FloatingIPsAssociationTestJSON-1881166845-project-member] [instance: 3a848010-382b-4753-8420-37222963e7e2] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 627.466593] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] [instance: c7670280-b5a7-4de2-8cf9-6814a81eb1a6] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 4a3a996d-9a97-4d00-a63e-71097cd065b2, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance c7670280-b5a7-4de2-8cf9-6814a81eb1a6 was re-scheduled: Binding failed for port 4a3a996d-9a97-4d00-a63e-71097cd065b2, please check neutron logs for more information.\n'] [ 627.466593] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 627.466593] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c7670280-b5a7-4de2-8cf9-6814a81eb1a6.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c7670280-b5a7-4de2-8cf9-6814a81eb1a6. [ 627.466593] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] [instance: c7670280-b5a7-4de2-8cf9-6814a81eb1a6] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c7670280-b5a7-4de2-8cf9-6814a81eb1a6. [ 627.496928] nova-conductor[54692]: DEBUG nova.network.neutron [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] [instance: c7670280-b5a7-4de2-8cf9-6814a81eb1a6] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 627.514724] nova-conductor[54692]: DEBUG nova.network.neutron [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] [instance: c7670280-b5a7-4de2-8cf9-6814a81eb1a6] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 627.518832] nova-conductor[54692]: DEBUG nova.network.neutron [None req-eec3f13e-5e2e-472a-bf7d-0b17670f6a28 tempest-ServersTestManualDisk-1672934667 tempest-ServersTestManualDisk-1672934667-project-member] [instance: c7670280-b5a7-4de2-8cf9-6814a81eb1a6] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 628.281040] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Expecting reply to msg 105df1ad6b7244bd8982e9fb1e537e13 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 628.496502] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 105df1ad6b7244bd8982e9fb1e537e13 [ 628.798755] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Took 0.51 seconds to select destinations for 2 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 628.817755] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 628.818098] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 628.818377] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 628.851435] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 628.851663] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 628.851827] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 628.879916] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 628.880163] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 628.880354] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 628.880745] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 628.880932] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 628.881091] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 628.890775] nova-conductor[54692]: DEBUG nova.quota [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Getting quotas for project 6a34141edbd4482d9f709ad211179a90. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 628.893931] nova-conductor[54692]: DEBUG nova.quota [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Getting quotas for user ce8f935dca594cfa87b680e92f1c492c and project 6a34141edbd4482d9f709ad211179a90. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 628.900117] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: d633de23-23ab-4297-b2d6-2702906b31a9] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 628.900593] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 628.900796] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 628.900956] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 628.904544] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: d633de23-23ab-4297-b2d6-2702906b31a9] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 628.905184] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 628.905377] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 628.905536] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 628.918633] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 628.918848] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 628.919010] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 628.925435] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: 88c5a4ae-ab48-4873-8a42-ecc6d0f99dd0] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 628.926191] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 628.926191] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 628.926448] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 628.937430] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: 88c5a4ae-ab48-4873-8a42-ecc6d0f99dd0] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 628.937812] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 628.937959] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 628.938133] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 628.954239] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 628.954482] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 628.955011] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 632.288341] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=377,nova_cell1:UPDATE=188,nova_cell1:SAVEPOINT=36,nova_cell1:RELEASE=36,nova_cell1:INSERT=30,nova_cell1:DELETE=2 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 632.560500] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Expecting reply to msg e7e651f3beb14f698a250078082c6fc5 in queue reply_4316ed433d59463c9660e50df524570b [ 632.710259] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg e7e651f3beb14f698a250078082c6fc5 [ 633.063817] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 633.079378] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 633.079680] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 633.079887] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 633.122882] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 633.122882] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 633.122882] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 633.122882] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 633.122882] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 633.122882] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 633.132482] nova-conductor[54691]: DEBUG nova.quota [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Getting quotas for project 82ebdd23e4964eedba2d5a377c62c702. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 633.136747] nova-conductor[54691]: DEBUG nova.quota [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Getting quotas for user fe9f67d1693249379c2fba512b44506c and project 82ebdd23e4964eedba2d5a377c62c702. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 633.151520] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] [instance: bbc25649-cfa4-4f5f-ab1e-5f1347780612] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 633.151993] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 633.152290] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 633.152417] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 633.164916] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] [instance: bbc25649-cfa4-4f5f-ab1e-5f1347780612] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 633.165859] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 633.165937] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 633.166102] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 633.181851] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 633.182077] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 633.182247] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3ae0a5db-433b-4be8-b161-03d7896dc541 tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 637.001296] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:SELECT=368,nova_cell1:UPDATE=171,nova_cell1:INSERT=23,nova_cell1:SAVEPOINT=35,nova_cell1:RELEASE=35 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 638.884446] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell0:SELECT=2 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 638.940492] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=8,nova_cell1:SAVEPOINT=2,nova_cell1:INSERT=23,nova_cell1:RELEASE=2 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 638.950092] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=14,nova_api:DELETE=2,nova_api:UPDATE=2 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 638.952247] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=14,nova_api:UPDATE=1,nova_api:DELETE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 642.308834] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Expecting reply to msg 3cdd55dd709b435ba88ef35ee415c8fa in queue reply_4316ed433d59463c9660e50df524570b [ 642.492645] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 3cdd55dd709b435ba88ef35ee415c8fa [ 642.818565] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 642.836242] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 642.836242] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 642.836242] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 642.889065] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 642.889065] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 642.889065] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 642.889065] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 642.889065] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 642.889065] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 642.906357] nova-conductor[54691]: DEBUG nova.quota [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Getting quotas for project 7c343033abe74ef28f09e614ffad9254. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 642.909584] nova-conductor[54691]: DEBUG nova.quota [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Getting quotas for user 613209c7c103464ea473f32bcf0116b1 and project 7c343033abe74ef28f09e614ffad9254. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 642.915027] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] [instance: 6a7e7ba2-a972-48bd-83b7-8d1b6a3289ca] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 642.915544] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 642.915754] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 642.915912] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 642.921440] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] [instance: 6a7e7ba2-a972-48bd-83b7-8d1b6a3289ca] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 642.922150] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 642.922657] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 642.922751] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 642.942604] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 642.943077] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 642.943205] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 643.441850] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Expecting reply to msg da094b55c2b84782a0ebe198f2d617a3 in queue reply_4316ed433d59463c9660e50df524570b [ 643.588259] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg da094b55c2b84782a0ebe198f2d617a3 [ 643.798125] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Expecting reply to msg ff4c19503f68429d81a8a036396eecc7 in queue reply_4316ed433d59463c9660e50df524570b [ 643.947710] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 643.958899] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg ff4c19503f68429d81a8a036396eecc7 [ 643.960982] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 643.961416] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 643.961788] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 644.025701] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 644.026193] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 644.026559] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 644.027154] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 644.027538] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 644.027942] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 644.054437] nova-conductor[54691]: DEBUG nova.quota [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Getting quotas for project eb7d255444a54cc3b2f7ede00ec3bbca. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 644.062349] nova-conductor[54691]: DEBUG nova.quota [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Getting quotas for user 1a48bd168e18415d8fca989071e048d3 and project eb7d255444a54cc3b2f7ede00ec3bbca. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 644.068885] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] [instance: 3135faf8-b918-42f5-bcaa-c23a19166bb3] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 644.070134] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 644.070527] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 644.070866] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 644.075980] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] [instance: 3135faf8-b918-42f5-bcaa-c23a19166bb3] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 644.076878] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 644.077229] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 644.077546] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 644.093584] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 644.094006] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 644.094342] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 644.291047] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Expecting reply to msg 6ea5f6a23f324ca6b2f7b97322cd1a64 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 644.301750] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 644.314855] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 644.315074] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 644.315240] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 644.360802] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 644.361242] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 644.361748] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 644.362310] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 644.362627] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 644.362926] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 644.373086] nova-conductor[54691]: DEBUG nova.quota [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Getting quotas for project dc923c7bf786472c9d72f276a49e29fb. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 644.375650] nova-conductor[54691]: DEBUG nova.quota [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Getting quotas for user fe5ca027bccd496bbfb9957653cb612d and project dc923c7bf786472c9d72f276a49e29fb. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 644.390931] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] [instance: 97927074-a676-4ebd-aa7a-fe92e3ce1be6] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 644.391650] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 644.392104] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 644.392387] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 644.402587] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] [instance: 97927074-a676-4ebd-aa7a-fe92e3ce1be6] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 644.403372] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 644.403683] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 644.403945] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 644.426595] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 644.427310] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 644.428148] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 644.461668] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 6ea5f6a23f324ca6b2f7b97322cd1a64 [ 644.793759] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 644.807499] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 644.807679] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 644.807851] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 644.843709] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 644.843932] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 644.844113] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 644.844485] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 644.844668] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 644.844863] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 644.869228] nova-conductor[54692]: DEBUG nova.quota [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Getting quotas for project dc923c7bf786472c9d72f276a49e29fb. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 644.873896] nova-conductor[54692]: DEBUG nova.quota [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Getting quotas for user fe5ca027bccd496bbfb9957653cb612d and project dc923c7bf786472c9d72f276a49e29fb. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 644.879838] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] [instance: 61f31838-2fd1-4804-82b7-56b3a08c3ff5] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 644.880288] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 644.880490] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 644.880652] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 644.887147] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] [instance: 61f31838-2fd1-4804-82b7-56b3a08c3ff5] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 644.887822] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 644.888036] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 644.888203] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 644.907572] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 644.907794] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 644.907955] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 645.049948] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: 558058e0-9808-415a-8866-75d1c8598d43] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port fa29c075-7ce5-482f-8388-05891144179b, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 558058e0-9808-415a-8866-75d1c8598d43 was re-scheduled: Binding failed for port fa29c075-7ce5-482f-8388-05891144179b, please check neutron logs for more information.\n'] [ 645.051099] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 645.051475] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 558058e0-9808-415a-8866-75d1c8598d43.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 558058e0-9808-415a-8866-75d1c8598d43. [ 645.053278] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: 558058e0-9808-415a-8866-75d1c8598d43] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 558058e0-9808-415a-8866-75d1c8598d43. [ 645.086361] nova-conductor[54691]: DEBUG nova.network.neutron [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: 558058e0-9808-415a-8866-75d1c8598d43] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 645.149853] nova-conductor[54691]: DEBUG nova.network.neutron [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: 558058e0-9808-415a-8866-75d1c8598d43] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 645.153133] nova-conductor[54691]: DEBUG nova.network.neutron [None req-b1c7ae72-5848-4021-afd4-652131ec82bd tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: 558058e0-9808-415a-8866-75d1c8598d43] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 645.842848] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Expecting reply to msg 9a207943e23d4e18b9e681669d450f5d in queue reply_4316ed433d59463c9660e50df524570b [ 645.967869] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 9a207943e23d4e18b9e681669d450f5d [ 646.310148] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-0b3f5c14-eda3-4bde-8327-dd5c1fa471e4 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Targeting cell 2d4e634d-6292-45c4-9eb1-09b106cacc48(cell1) for conductor method rebuild_instance {{(pid=54692) wrapper /opt/stack/nova/nova/conductor/manager.py:95}} [ 646.310419] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-0b3f5c14-eda3-4bde-8327-dd5c1fa471e4 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 646.310649] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-0b3f5c14-eda3-4bde-8327-dd5c1fa471e4 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 646.310820] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-0b3f5c14-eda3-4bde-8327-dd5c1fa471e4 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 646.327923] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-0b3f5c14-eda3-4bde-8327-dd5c1fa471e4 tempest-ServerShowV247Test-815835844 tempest-ServerShowV247Test-815835844-project-member] [instance: 13247e87-08ab-434e-b091-8b6116280fb7] No migration record for the rebuild/evacuate request. {{(pid=54692) rebuild_instance /opt/stack/nova/nova/conductor/manager.py:1227}} [ 646.345765] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 646.361957] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 646.362187] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 646.362353] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 646.389580] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 646.389803] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 646.389965] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 646.390338] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 646.390533] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 646.390665] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 646.398644] nova-conductor[54691]: DEBUG nova.quota [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Getting quotas for project de11ac365f704c2986c782a5918f802f. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 646.400864] nova-conductor[54691]: DEBUG nova.quota [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Getting quotas for user c43c33165931465494c0a7da171126e4 and project de11ac365f704c2986c782a5918f802f. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 646.405770] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: e7b5d223-a1c7-4b17-a14b-9d75deeb3ac0] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 646.406219] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 646.406421] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 646.406589] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 646.409636] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: e7b5d223-a1c7-4b17-a14b-9d75deeb3ac0] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 646.410262] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 646.410481] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 646.410648] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 646.424662] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 646.424870] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 646.425032] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 654.848654] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell0:SELECT=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 654.906158] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=6,nova_api:DELETE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 655.723500] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] [instance: a3182b4a-1846-4766-8988-aab0a74fa4c8] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 0b41917f-afed-4e53-b215-68a6e91bbd7d, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance a3182b4a-1846-4766-8988-aab0a74fa4c8 was re-scheduled: Binding failed for port 0b41917f-afed-4e53-b215-68a6e91bbd7d, please check neutron logs for more information.\n'] [ 655.724140] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 655.724383] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance a3182b4a-1846-4766-8988-aab0a74fa4c8.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance a3182b4a-1846-4766-8988-aab0a74fa4c8. [ 655.724699] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] [instance: a3182b4a-1846-4766-8988-aab0a74fa4c8] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance a3182b4a-1846-4766-8988-aab0a74fa4c8. [ 655.750075] nova-conductor[54691]: DEBUG nova.network.neutron [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] [instance: a3182b4a-1846-4766-8988-aab0a74fa4c8] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 655.767085] nova-conductor[54691]: DEBUG nova.network.neutron [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] [instance: a3182b4a-1846-4766-8988-aab0a74fa4c8] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 655.770349] nova-conductor[54691]: DEBUG nova.network.neutron [None req-2f447a51-dc80-4fa4-a769-be1c0cdb279c tempest-ServerDiagnosticsTest-2135904566 tempest-ServerDiagnosticsTest-2135904566-project-member] [instance: a3182b4a-1846-4766-8988-aab0a74fa4c8] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 656.330121] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=8,nova_api:UPDATE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 656.344367] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=7,nova_cell1:INSERT=8,nova_cell1:UPDATE=3 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 656.395186] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell0:SELECT=5 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 656.412402] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:SELECT=15,nova_cell1:SAVEPOINT=4,nova_cell1:INSERT=39,nova_cell1:RELEASE=4 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 656.418635] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=26,nova_api:DELETE=2,nova_api:UPDATE=2 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 656.423633] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=34,nova_api:UPDATE=3,nova_api:DELETE=3 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 657.704716] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] [instance: c97b25ac-527f-404e-a539-1c0f43bcd66b] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port f755d6f1-e7ae-4c6f-9417-ff0e25adb144, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance c97b25ac-527f-404e-a539-1c0f43bcd66b was re-scheduled: Binding failed for port f755d6f1-e7ae-4c6f-9417-ff0e25adb144, please check neutron logs for more information.\n'] [ 657.705298] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 657.705541] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c97b25ac-527f-404e-a539-1c0f43bcd66b.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c97b25ac-527f-404e-a539-1c0f43bcd66b. [ 657.705865] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] [instance: c97b25ac-527f-404e-a539-1c0f43bcd66b] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c97b25ac-527f-404e-a539-1c0f43bcd66b. [ 657.737410] nova-conductor[54692]: DEBUG nova.network.neutron [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] [instance: c97b25ac-527f-404e-a539-1c0f43bcd66b] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 657.754410] nova-conductor[54692]: DEBUG nova.network.neutron [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] [instance: c97b25ac-527f-404e-a539-1c0f43bcd66b] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 657.758053] nova-conductor[54692]: DEBUG nova.network.neutron [None req-7f2ca26b-ac76-4032-b1f1-602304d45f27 tempest-DeleteServersAdminTestJSON-1916310689 tempest-DeleteServersAdminTestJSON-1916310689-project-member] [instance: c97b25ac-527f-404e-a539-1c0f43bcd66b] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 658.728981] nova-conductor[54692]: DEBUG nova.db.main.api [None req-4a1eeeca-c204-444f-801f-aa11d372fc55 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Created instance_extra for c4512101-c88c-4f3c-bde1-def0f3520101 {{(pid=54692) instance_extra_update_by_uuid /opt/stack/nova/nova/db/main/api.py:2551}} [ 659.746013] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] [instance: 4e2f1fc0-102f-4d39-bec2-a2651918a202] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 48e4b255-39f8-4691-be1e-141dd9425261, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 4e2f1fc0-102f-4d39-bec2-a2651918a202 was re-scheduled: Binding failed for port 48e4b255-39f8-4691-be1e-141dd9425261, please check neutron logs for more information.\n'] [ 659.747193] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 659.747654] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 4e2f1fc0-102f-4d39-bec2-a2651918a202.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 4e2f1fc0-102f-4d39-bec2-a2651918a202. [ 659.747829] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] [instance: 4e2f1fc0-102f-4d39-bec2-a2651918a202] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 4e2f1fc0-102f-4d39-bec2-a2651918a202. [ 659.785678] nova-conductor[54692]: DEBUG nova.network.neutron [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] [instance: 4e2f1fc0-102f-4d39-bec2-a2651918a202] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 659.824204] nova-conductor[54692]: DEBUG nova.network.neutron [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] [instance: 4e2f1fc0-102f-4d39-bec2-a2651918a202] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 659.830105] nova-conductor[54692]: DEBUG nova.network.neutron [None req-f76d2fd6-064e-4a57-aba4-a3b551967109 tempest-ServerActionsV293TestJSON-1135923703 tempest-ServerActionsV293TestJSON-1135923703-project-member] [instance: 4e2f1fc0-102f-4d39-bec2-a2651918a202] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 661.393055] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Expecting reply to msg 0b9d0f5cb2cb4d05abdb1e3571966119 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 661.521883] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 0b9d0f5cb2cb4d05abdb1e3571966119 [ 661.895364] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 661.909177] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 661.909566] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 661.909621] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 661.962835] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 661.963076] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 661.963243] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 661.963662] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 661.963852] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 661.964029] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 661.973624] nova-conductor[54692]: DEBUG nova.quota [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Getting quotas for project 8003d16b1cc048bc94407d028838a4f6. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 661.976692] nova-conductor[54692]: DEBUG nova.quota [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Getting quotas for user cd65b3ab163145e1bf370aa0dc63f65c and project 8003d16b1cc048bc94407d028838a4f6. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 661.981987] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] [instance: fd2547e6-9449-4839-8ebb-f02f01a85b8c] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 661.982419] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 661.982626] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 661.982789] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 661.987274] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] [instance: fd2547e6-9449-4839-8ebb-f02f01a85b8c] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 661.987921] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 661.988139] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 661.988306] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 662.004648] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 662.004873] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 662.005040] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 664.400251] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] [instance: ce17af27-45c3-461e-8cb7-b9b023d88361] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 08f341c7-922c-41cc-8984-e8f0675c276f, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance ce17af27-45c3-461e-8cb7-b9b023d88361 was re-scheduled: Binding failed for port 08f341c7-922c-41cc-8984-e8f0675c276f, please check neutron logs for more information.\n'] [ 664.400902] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 664.401113] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ce17af27-45c3-461e-8cb7-b9b023d88361.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ce17af27-45c3-461e-8cb7-b9b023d88361. [ 664.401329] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] [instance: ce17af27-45c3-461e-8cb7-b9b023d88361] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ce17af27-45c3-461e-8cb7-b9b023d88361. [ 664.428111] nova-conductor[54691]: DEBUG nova.network.neutron [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] [instance: ce17af27-45c3-461e-8cb7-b9b023d88361] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 664.456405] nova-conductor[54691]: DEBUG nova.network.neutron [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] [instance: ce17af27-45c3-461e-8cb7-b9b023d88361] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 664.460879] nova-conductor[54691]: DEBUG nova.network.neutron [None req-ddb16295-9cca-498a-bd70-98ac0aa624b9 tempest-ServerAddressesNegativeTestJSON-2070091609 tempest-ServerAddressesNegativeTestJSON-2070091609-project-member] [instance: ce17af27-45c3-461e-8cb7-b9b023d88361] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 671.004695] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Expecting reply to msg cb52920639534f289d58e0cb2eb6829a in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 671.159087] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg cb52920639534f289d58e0cb2eb6829a [ 671.508568] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 671.520718] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 671.521152] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 671.521398] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 671.553286] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 671.553569] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 671.553794] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 671.554258] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 671.554515] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 671.554729] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 671.569859] nova-conductor[54692]: DEBUG nova.quota [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Getting quotas for project 9a349372836346d8983c95b50d980e37. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 671.572018] nova-conductor[54692]: DEBUG nova.quota [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Getting quotas for user 23c687bad35b409ab96bfd7cde54fd27 and project 9a349372836346d8983c95b50d980e37. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 671.580416] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] [instance: 5014a870-32fd-4166-9446-c41ed2a4954e] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 671.580416] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 671.580416] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 671.580416] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 671.584770] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] [instance: 5014a870-32fd-4166-9446-c41ed2a4954e] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 671.586031] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 671.586497] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 671.586812] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 671.610137] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] [instance: 9a445d5c-7a5f-4149-b198-306a04b4c040] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 012ccbce-db53-42bf-b39c-ede4b3b9b3fc, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 9a445d5c-7a5f-4149-b198-306a04b4c040 was re-scheduled: Binding failed for port 012ccbce-db53-42bf-b39c-ede4b3b9b3fc, please check neutron logs for more information.\n'] [ 671.611424] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 671.611816] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9a445d5c-7a5f-4149-b198-306a04b4c040.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9a445d5c-7a5f-4149-b198-306a04b4c040. [ 671.612366] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] [instance: 9a445d5c-7a5f-4149-b198-306a04b4c040] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9a445d5c-7a5f-4149-b198-306a04b4c040. [ 671.641919] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 671.642144] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 671.642316] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 671.652147] nova-conductor[54692]: DEBUG nova.network.neutron [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] [instance: 9a445d5c-7a5f-4149-b198-306a04b4c040] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 671.669185] nova-conductor[54692]: DEBUG nova.network.neutron [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] [instance: 9a445d5c-7a5f-4149-b198-306a04b4c040] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 671.673116] nova-conductor[54692]: DEBUG nova.network.neutron [None req-04f976f4-2b10-4abf-ac24-b458e43865bc tempest-ImagesOneServerTestJSON-1857740873 tempest-ImagesOneServerTestJSON-1857740873-project-member] [instance: 9a445d5c-7a5f-4149-b198-306a04b4c040] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 676.561202] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Expecting reply to msg e48ae3e85b4c466799b0eba4d7fc0cb7 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 676.705349] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg e48ae3e85b4c466799b0eba4d7fc0cb7 [ 677.066628] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 677.081702] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 677.082117] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.002s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 677.082420] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 677.118380] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 677.118822] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 677.119165] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 677.119693] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 677.119991] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 677.120317] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 677.129551] nova-conductor[54692]: DEBUG nova.quota [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Getting quotas for project 6362a2ff1b4046cb981b9552a7f621b1. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 677.132714] nova-conductor[54692]: DEBUG nova.quota [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Getting quotas for user ce05881c446e44ee9ca81e5416956ebb and project 6362a2ff1b4046cb981b9552a7f621b1. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 677.137450] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] [instance: b49cde53-0ef7-44a4-b894-e37d1cbf41a7] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 677.138007] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 677.138310] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 677.138584] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 677.143316] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] [instance: b49cde53-0ef7-44a4-b894-e37d1cbf41a7] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 677.144100] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 677.144400] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 677.144687] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 677.161237] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 677.161569] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 677.161839] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 687.125036] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell0:SELECT=3 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 687.146943] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=9,nova_cell1:SAVEPOINT=3,nova_cell1:INSERT=24,nova_cell1:RELEASE=3 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 687.156715] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=17,nova_api:UPDATE=2,nova_api:DELETE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 687.159032] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=19,nova_api:DELETE=2,nova_api:UPDATE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 687.367897] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] [instance: ad008294-52db-40de-95ad-3e0f2d30a462] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 8a271f17-c894-4132-ac3d-e76ca3e8fbfa, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance ad008294-52db-40de-95ad-3e0f2d30a462 was re-scheduled: Binding failed for port 8a271f17-c894-4132-ac3d-e76ca3e8fbfa, please check neutron logs for more information.\n'] [ 687.368337] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 687.368651] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ad008294-52db-40de-95ad-3e0f2d30a462.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ad008294-52db-40de-95ad-3e0f2d30a462. [ 687.368965] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] [instance: ad008294-52db-40de-95ad-3e0f2d30a462] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ad008294-52db-40de-95ad-3e0f2d30a462. [ 687.389145] nova-conductor[54692]: DEBUG nova.network.neutron [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] [instance: ad008294-52db-40de-95ad-3e0f2d30a462] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 687.404972] nova-conductor[54692]: DEBUG nova.network.neutron [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] [instance: ad008294-52db-40de-95ad-3e0f2d30a462] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 687.408026] nova-conductor[54692]: DEBUG nova.network.neutron [None req-65ab93c4-5104-4f28-8ec5-d058d5e89d95 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] [instance: ad008294-52db-40de-95ad-3e0f2d30a462] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 689.971915] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Expecting reply to msg 64280ba58f4a4db0b37fbef3b72bb07a in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 690.112198] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 64280ba58f4a4db0b37fbef3b72bb07a [ 690.257847] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] [instance: 38767b46-9a31-4031-981b-424497dc4a34] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 04b56e77-5706-4977-9ead-92b48295fcb5, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 38767b46-9a31-4031-981b-424497dc4a34 was re-scheduled: Binding failed for port 04b56e77-5706-4977-9ead-92b48295fcb5, please check neutron logs for more information.\n'] [ 690.258434] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 690.258667] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 38767b46-9a31-4031-981b-424497dc4a34.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 38767b46-9a31-4031-981b-424497dc4a34. [ 690.260829] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] [instance: 38767b46-9a31-4031-981b-424497dc4a34] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 38767b46-9a31-4031-981b-424497dc4a34. [ 690.285948] nova-conductor[54692]: DEBUG nova.network.neutron [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] [instance: 38767b46-9a31-4031-981b-424497dc4a34] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 690.305334] nova-conductor[54692]: DEBUG nova.network.neutron [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] [instance: 38767b46-9a31-4031-981b-424497dc4a34] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 690.309733] nova-conductor[54692]: DEBUG nova.network.neutron [None req-0ec7a21f-d5ff-4ccf-8039-bee024fa795d tempest-InstanceActionsV221TestJSON-646378659 tempest-InstanceActionsV221TestJSON-646378659-project-member] [instance: 38767b46-9a31-4031-981b-424497dc4a34] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 690.474076] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 690.504350] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 690.505294] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 690.505982] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 690.534206] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 690.534435] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 690.534608] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 690.534977] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 690.535329] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 690.535329] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 690.542776] nova-conductor[54692]: DEBUG nova.quota [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Getting quotas for project d781d0a045444391bc070520be1b6431. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 690.549247] nova-conductor[54692]: DEBUG nova.quota [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Getting quotas for user 6a6d2a917fec48e48e943137dff043fd and project d781d0a045444391bc070520be1b6431. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 690.558351] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] [instance: 25e906f6-ec98-4c99-83c7-43f94c5790dd] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 690.558844] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 690.559061] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 690.559228] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 690.564421] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] [instance: 25e906f6-ec98-4c99-83c7-43f94c5790dd] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 690.564421] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 690.564421] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 690.564421] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 690.585031] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 690.585031] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 690.585110] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 692.573950] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:UPDATE=138,nova_cell1:SELECT=304,nova_cell1:SAVEPOINT=30,nova_cell1:RELEASE=30,nova_cell1:INSERT=24 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 696.342930] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Expecting reply to msg 58745057a12c4f11940c3b3b78771bf3 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 696.474131] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 58745057a12c4f11940c3b3b78771bf3 [ 696.846429] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Expecting reply to msg 92bc514ddebd4560a5991386941e3b6a in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 696.846731] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 696.864129] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 696.864421] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 696.864645] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 696.897192] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 696.897591] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 696.897774] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 696.898378] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 696.898606] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 696.898823] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 696.908677] nova-conductor[54692]: DEBUG nova.quota [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Getting quotas for project 9703f47d3d944740a48535c270de37ed. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 696.910109] nova-conductor[54692]: DEBUG nova.quota [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Getting quotas for user 303068b2037f422b9cba37aebaca71ab and project 9703f47d3d944740a48535c270de37ed. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 696.915213] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] [instance: 99735498-3c85-4a01-bc05-4def014618a8] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 696.915754] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 696.916025] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 696.916233] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 696.920735] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] [instance: 99735498-3c85-4a01-bc05-4def014618a8] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 696.921570] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 696.921744] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 696.921949] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 696.940225] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 696.940518] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 696.940913] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 696.987716] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 92bc514ddebd4560a5991386941e3b6a [ 697.017172] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:SELECT=321,nova_cell1:UPDATE=164,nova_cell1:SAVEPOINT=28,nova_cell1:RELEASE=28,nova_cell1:INSERT=19,nova_cell1:DELETE=2 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 697.350943] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 697.366183] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 697.366402] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 697.366569] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 697.413691] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 697.413914] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 697.414087] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 697.414433] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 697.414610] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 697.414761] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 697.422433] nova-conductor[54692]: DEBUG nova.quota [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Getting quotas for project 9703f47d3d944740a48535c270de37ed. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 697.424733] nova-conductor[54692]: DEBUG nova.quota [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Getting quotas for user 303068b2037f422b9cba37aebaca71ab and project 9703f47d3d944740a48535c270de37ed. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 697.429949] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] [instance: 6c4ac72f-1636-4f4c-928c-0a6fe895ce37] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 697.430477] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 697.430680] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 697.430853] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 697.434145] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] [instance: 6c4ac72f-1636-4f4c-928c-0a6fe895ce37] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 697.434773] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 697.434961] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 697.435117] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 697.447996] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 697.448306] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 697.448474] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 699.717830] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] [instance: 6480b28b-7ded-46c1-8fa0-6e1abba4bf7a] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 32038c9e-fd2a-4e52-9f5c-d9dc607315cb, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 6480b28b-7ded-46c1-8fa0-6e1abba4bf7a was re-scheduled: Binding failed for port 32038c9e-fd2a-4e52-9f5c-d9dc607315cb, please check neutron logs for more information.\n'] [ 699.718396] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 699.718618] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 6480b28b-7ded-46c1-8fa0-6e1abba4bf7a.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 6480b28b-7ded-46c1-8fa0-6e1abba4bf7a. [ 699.718819] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] [instance: 6480b28b-7ded-46c1-8fa0-6e1abba4bf7a] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 6480b28b-7ded-46c1-8fa0-6e1abba4bf7a. [ 699.743291] nova-conductor[54691]: DEBUG nova.network.neutron [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] [instance: 6480b28b-7ded-46c1-8fa0-6e1abba4bf7a] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 699.760731] nova-conductor[54691]: DEBUG nova.network.neutron [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] [instance: 6480b28b-7ded-46c1-8fa0-6e1abba4bf7a] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 699.764931] nova-conductor[54691]: DEBUG nova.network.neutron [None req-1f17e573-26e4-4b7a-9707-f4de1683531c tempest-ServerPasswordTestJSON-873993815 tempest-ServerPasswordTestJSON-873993815-project-member] [instance: 6480b28b-7ded-46c1-8fa0-6e1abba4bf7a] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 704.287819] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] [instance: 41650a98-fda7-44fd-aca6-d3b94cf17308] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 0718b236-24c2-4f9c-b72f-49817b897ae7, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 41650a98-fda7-44fd-aca6-d3b94cf17308 was re-scheduled: Binding failed for port 0718b236-24c2-4f9c-b72f-49817b897ae7, please check neutron logs for more information.\n'] [ 704.289308] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 704.290829] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 41650a98-fda7-44fd-aca6-d3b94cf17308.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 41650a98-fda7-44fd-aca6-d3b94cf17308. [ 704.290829] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] [instance: 41650a98-fda7-44fd-aca6-d3b94cf17308] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 41650a98-fda7-44fd-aca6-d3b94cf17308. [ 704.314387] nova-conductor[54691]: DEBUG nova.network.neutron [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] [instance: 41650a98-fda7-44fd-aca6-d3b94cf17308] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 704.333184] nova-conductor[54691]: DEBUG nova.network.neutron [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] [instance: 41650a98-fda7-44fd-aca6-d3b94cf17308] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 704.336405] nova-conductor[54691]: DEBUG nova.network.neutron [None req-8e565307-45d9-4339-abba-636f1e61428e tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] [instance: 41650a98-fda7-44fd-aca6-d3b94cf17308] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 706.361886] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] [instance: 3e82fa5d-6610-4d3d-972f-4120aca5e8a7] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 76838e00-f173-4ff1-94b4-a3c1022b850e, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 3e82fa5d-6610-4d3d-972f-4120aca5e8a7 was re-scheduled: Binding failed for port 76838e00-f173-4ff1-94b4-a3c1022b850e, please check neutron logs for more information.\n'] [ 706.362479] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 706.363118] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 3e82fa5d-6610-4d3d-972f-4120aca5e8a7.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 3e82fa5d-6610-4d3d-972f-4120aca5e8a7. [ 706.363118] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] [instance: 3e82fa5d-6610-4d3d-972f-4120aca5e8a7] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 3e82fa5d-6610-4d3d-972f-4120aca5e8a7. [ 706.383151] nova-conductor[54692]: DEBUG nova.network.neutron [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] [instance: 3e82fa5d-6610-4d3d-972f-4120aca5e8a7] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 706.406289] nova-conductor[54692]: DEBUG nova.network.neutron [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] [instance: 3e82fa5d-6610-4d3d-972f-4120aca5e8a7] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 706.409466] nova-conductor[54692]: DEBUG nova.network.neutron [None req-8dd224a5-7c1b-4ae1-b3b5-fece2290ac2d tempest-AttachInterfacesV270Test-865365601 tempest-AttachInterfacesV270Test-865365601-project-member] [instance: 3e82fa5d-6610-4d3d-972f-4120aca5e8a7] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 707.418946] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell0:SELECT=3 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 707.436714] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=9,nova_cell1:INSERT=22,nova_cell1:SAVEPOINT=1,nova_cell1:RELEASE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 707.443450] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=20,nova_api:DELETE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 707.446042] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=16,nova_api:UPDATE=3,nova_api:DELETE=2 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 709.700595] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Expecting reply to msg beac051b90fc4fe5ade8cd0acfe66566 in queue reply_4316ed433d59463c9660e50df524570b [ 709.875394] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg beac051b90fc4fe5ade8cd0acfe66566 [ 710.208641] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 710.232041] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 710.232041] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 710.232041] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 710.268974] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 710.269419] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 710.269708] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 710.270197] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 710.270513] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 710.270794] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 710.279323] nova-conductor[54691]: DEBUG nova.quota [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Getting quotas for project 532d5738736643d2836c0896012494da. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 710.282490] nova-conductor[54691]: DEBUG nova.quota [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Getting quotas for user 16306c5f0e4d4eabbcce4a9c72edce5f and project 532d5738736643d2836c0896012494da. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 710.287244] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] [instance: 451b9e70-f202-4a10-8b19-34b6167ef14d] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 710.287782] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 710.288105] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 710.288405] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 710.295978] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] [instance: 451b9e70-f202-4a10-8b19-34b6167ef14d] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 710.296793] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 710.297099] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 710.297361] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 710.313937] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 710.314164] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 710.314336] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 713.874755] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Expecting reply to msg f176a0fada8c4f77af76d96d7699547b in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 713.997061] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg f176a0fada8c4f77af76d96d7699547b [ 714.377223] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 714.389092] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 714.389310] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 714.389502] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 714.424502] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 714.424730] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 714.424891] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 714.425281] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 714.425467] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 714.425624] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 714.432757] nova-conductor[54692]: DEBUG nova.quota [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Getting quotas for project 77e1b9d5d344410899f56a473155e5b2. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 714.434807] nova-conductor[54692]: DEBUG nova.quota [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Getting quotas for user 8d242f77c562441e82ccfb15eb0055b2 and project 77e1b9d5d344410899f56a473155e5b2. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 714.440946] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] [instance: 5d2465e7-e473-46e9-a8ce-cc58dcdbd28f] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 714.441377] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 714.441572] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 714.441734] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 714.449590] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] [instance: 5d2465e7-e473-46e9-a8ce-cc58dcdbd28f] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 714.450252] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 714.450452] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 714.450616] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 714.463558] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 714.463779] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 714.463967] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8a9891a5-18b4-49d2-9051-d0466b6843da tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 717.755323] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] [instance: 15215512-da0c-407f-a352-43c051698f2d] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 56565dec-4584-4838-afdb-14c02b5da697, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 15215512-da0c-407f-a352-43c051698f2d was re-scheduled: Binding failed for port 56565dec-4584-4838-afdb-14c02b5da697, please check neutron logs for more information.\n'] [ 717.755952] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 717.756211] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 15215512-da0c-407f-a352-43c051698f2d.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 15215512-da0c-407f-a352-43c051698f2d. [ 717.756516] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] [instance: 15215512-da0c-407f-a352-43c051698f2d] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 15215512-da0c-407f-a352-43c051698f2d. [ 717.776201] nova-conductor[54692]: DEBUG nova.network.neutron [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] [instance: 15215512-da0c-407f-a352-43c051698f2d] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 717.793840] nova-conductor[54692]: DEBUG nova.network.neutron [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] [instance: 15215512-da0c-407f-a352-43c051698f2d] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 717.797163] nova-conductor[54692]: DEBUG nova.network.neutron [None req-492d4bbb-c067-4c11-aff5-c1d7608618d0 tempest-ServersTestFqdnHostnames-893793326 tempest-ServersTestFqdnHostnames-893793326-project-member] [instance: 15215512-da0c-407f-a352-43c051698f2d] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 719.669443] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] [instance: b88d76a9-6a30-4d5f-a5cd-f698e005dab5] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port ed90fdc0-76f6-4c25-82fe-d55ce7fa35e2, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance b88d76a9-6a30-4d5f-a5cd-f698e005dab5 was re-scheduled: Binding failed for port ed90fdc0-76f6-4c25-82fe-d55ce7fa35e2, please check neutron logs for more information.\n'] [ 719.669443] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 719.669443] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance b88d76a9-6a30-4d5f-a5cd-f698e005dab5.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance b88d76a9-6a30-4d5f-a5cd-f698e005dab5. [ 719.669443] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] [instance: b88d76a9-6a30-4d5f-a5cd-f698e005dab5] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance b88d76a9-6a30-4d5f-a5cd-f698e005dab5. [ 719.704213] nova-conductor[54692]: DEBUG nova.network.neutron [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] [instance: b88d76a9-6a30-4d5f-a5cd-f698e005dab5] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 719.729490] nova-conductor[54692]: DEBUG nova.network.neutron [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] [instance: b88d76a9-6a30-4d5f-a5cd-f698e005dab5] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 719.732762] nova-conductor[54692]: DEBUG nova.network.neutron [None req-669cbdde-a42e-4038-873d-964c62b5e74a tempest-ServersAdminNegativeTestJSON-1339964137 tempest-ServersAdminNegativeTestJSON-1339964137-project-member] [instance: b88d76a9-6a30-4d5f-a5cd-f698e005dab5] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 720.275795] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell0:SELECT=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 720.300298] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:SELECT=3,nova_cell1:INSERT=7 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 720.312243] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=5 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 720.312243] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=7,nova_api:UPDATE=1,nova_api:DELETE=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 721.735331] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] [instance: eeb7cdc7-427f-496f-be5d-60151a4bb0f3] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port a20f45f8-34a8-433b-b086-d64d4a77b648, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance eeb7cdc7-427f-496f-be5d-60151a4bb0f3 was re-scheduled: Binding failed for port a20f45f8-34a8-433b-b086-d64d4a77b648, please check neutron logs for more information.\n'] [ 721.735912] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 721.736155] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance eeb7cdc7-427f-496f-be5d-60151a4bb0f3.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance eeb7cdc7-427f-496f-be5d-60151a4bb0f3. [ 721.736469] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] [instance: eeb7cdc7-427f-496f-be5d-60151a4bb0f3] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance eeb7cdc7-427f-496f-be5d-60151a4bb0f3. [ 721.771708] nova-conductor[54691]: DEBUG nova.network.neutron [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] [instance: eeb7cdc7-427f-496f-be5d-60151a4bb0f3] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 721.803020] nova-conductor[54691]: DEBUG nova.network.neutron [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] [instance: eeb7cdc7-427f-496f-be5d-60151a4bb0f3] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 721.807969] nova-conductor[54691]: DEBUG nova.network.neutron [None req-25baa20c-fe6e-4086-9fc3-4043d24bed79 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] [instance: eeb7cdc7-427f-496f-be5d-60151a4bb0f3] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 724.004702] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Expecting reply to msg ced346d243e74540ae41e53502345910 in queue reply_4316ed433d59463c9660e50df524570b [ 724.150716] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg ced346d243e74540ae41e53502345910 [ 724.428770] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell0:SELECT=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 724.453288] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=3,nova_cell1:SAVEPOINT=1,nova_cell1:INSERT=8,nova_cell1:RELEASE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 724.466651] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=8,nova_api:UPDATE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 724.467316] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=4,nova_api:DELETE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 724.507164] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 724.528989] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 724.529203] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 724.529466] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 724.583544] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 724.584471] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 724.584680] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 724.585860] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 724.586061] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 724.586241] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 724.605800] nova-conductor[54691]: DEBUG nova.quota [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Getting quotas for project 9df640c9ce4d4a7e8a953841d0912936. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 724.608969] nova-conductor[54691]: DEBUG nova.quota [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Getting quotas for user 9a71b1891f254423bd5e6bd2be1641ee and project 9df640c9ce4d4a7e8a953841d0912936. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 724.613869] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] [instance: ee85f964-db45-404c-9016-b618748ec63c] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 724.614306] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 724.614510] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 724.614669] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 724.620713] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] [instance: ee85f964-db45-404c-9016-b618748ec63c] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 724.621372] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 724.621567] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 724.621750] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 724.652036] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 724.652285] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 724.652570] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 726.257430] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] [instance: 1754ee22-ae95-4439-a6c0-200fd153bad0] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 9ba330ad-0f59-4b8a-898d-9edc8943371d, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 1754ee22-ae95-4439-a6c0-200fd153bad0 was re-scheduled: Binding failed for port 9ba330ad-0f59-4b8a-898d-9edc8943371d, please check neutron logs for more information.\n'] [ 726.258279] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 726.258627] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1754ee22-ae95-4439-a6c0-200fd153bad0.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1754ee22-ae95-4439-a6c0-200fd153bad0. [ 726.258909] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] [instance: 1754ee22-ae95-4439-a6c0-200fd153bad0] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1754ee22-ae95-4439-a6c0-200fd153bad0. [ 726.284412] nova-conductor[54692]: DEBUG nova.network.neutron [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] [instance: 1754ee22-ae95-4439-a6c0-200fd153bad0] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 726.311230] nova-conductor[54692]: DEBUG nova.network.neutron [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] [instance: 1754ee22-ae95-4439-a6c0-200fd153bad0] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 726.315212] nova-conductor[54692]: DEBUG nova.network.neutron [None req-c8f0ed63-f79a-46cd-8da1-198b55ae3004 tempest-ServersAdminTestJSON-1300144382 tempest-ServersAdminTestJSON-1300144382-project-member] [instance: 1754ee22-ae95-4439-a6c0-200fd153bad0] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 727.304358] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Expecting reply to msg 1d0e38616b5649fca129673f153cb896 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 727.454556] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 1d0e38616b5649fca129673f153cb896 [ 727.809593] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 727.824377] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 727.824714] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 727.824958] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 727.865254] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 727.865578] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 727.865818] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 727.866250] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 727.866498] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 727.866718] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 727.876047] nova-conductor[54692]: DEBUG nova.quota [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Getting quotas for project adf774b7c36f46219e7dfdec663fe624. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 727.881234] nova-conductor[54692]: DEBUG nova.quota [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Getting quotas for user 949bec663bc44ef3b1d3eb8c8f6912a9 and project adf774b7c36f46219e7dfdec663fe624. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 727.887283] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: d7947374-0e8f-4d00-a430-923f0f2bb858] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 727.887790] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 727.888093] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 727.888332] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 727.893288] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: d7947374-0e8f-4d00-a430-923f0f2bb858] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 727.894062] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 727.894301] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 727.894544] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 727.910095] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 727.910407] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.002s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 727.910672] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 733.244874] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: 71ce0048-90fb-460c-a099-7eff5c092d35] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 0cfd9e3f-2f54-4f3f-a19a-dbd3ef53fc25, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 71ce0048-90fb-460c-a099-7eff5c092d35 was re-scheduled: Binding failed for port 0cfd9e3f-2f54-4f3f-a19a-dbd3ef53fc25, please check neutron logs for more information.\n'] [ 733.245556] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 733.245788] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 71ce0048-90fb-460c-a099-7eff5c092d35.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 71ce0048-90fb-460c-a099-7eff5c092d35. [ 733.246141] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: 71ce0048-90fb-460c-a099-7eff5c092d35] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 71ce0048-90fb-460c-a099-7eff5c092d35. [ 733.279085] nova-conductor[54691]: DEBUG nova.network.neutron [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: 71ce0048-90fb-460c-a099-7eff5c092d35] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 733.300222] nova-conductor[54691]: DEBUG nova.network.neutron [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: 71ce0048-90fb-460c-a099-7eff5c092d35] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 733.303902] nova-conductor[54691]: DEBUG nova.network.neutron [None req-d144e025-e9a0-447c-8f76-08532ffe6d50 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: 71ce0048-90fb-460c-a099-7eff5c092d35] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 734.317688] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Expecting reply to msg af10e9f6fa274b57ba6f28970c7b8e04 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 734.455428] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg af10e9f6fa274b57ba6f28970c7b8e04 [ 734.595338] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell0:SELECT=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 734.624229] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:SELECT=3,nova_cell1:SAVEPOINT=1,nova_cell1:INSERT=8,nova_cell1:RELEASE=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 734.642082] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=8 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 734.647760] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=4,nova_api:UPDATE=1,nova_api:DELETE=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 734.820061] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 734.833193] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 734.833427] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 734.833590] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 734.899317] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 734.899583] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 734.899752] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 734.900146] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 734.900331] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 734.900487] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 734.908309] nova-conductor[54692]: DEBUG nova.quota [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Getting quotas for project 2cf54adae5eb4b4eb3f8cef36e2ff1e9. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 734.911447] nova-conductor[54692]: DEBUG nova.quota [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Getting quotas for user 09d6c42a5b174d3d8f4eab5ad842826f and project 2cf54adae5eb4b4eb3f8cef36e2ff1e9. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 734.916637] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] [instance: 76c79f9e-7ce6-4af9-b937-8e4bd32d39fe] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 734.917116] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 734.917344] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 734.917614] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 734.921876] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] [instance: 76c79f9e-7ce6-4af9-b937-8e4bd32d39fe] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 734.922525] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 734.922724] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 734.922887] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 734.940517] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 734.940751] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 734.940916] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 735.181321] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Expecting reply to msg 05bd5de325a146948b461855930946f6 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 735.209186] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] [instance: 20d9b891-9c2a-479c-8fd8-93405480dee4] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 135528e4-b55f-4aeb-a74c-3b85b2005c68, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 20d9b891-9c2a-479c-8fd8-93405480dee4 was re-scheduled: Binding failed for port 135528e4-b55f-4aeb-a74c-3b85b2005c68, please check neutron logs for more information.\n'] [ 735.211293] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 735.211536] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 20d9b891-9c2a-479c-8fd8-93405480dee4.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 20d9b891-9c2a-479c-8fd8-93405480dee4. [ 735.211774] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] [instance: 20d9b891-9c2a-479c-8fd8-93405480dee4] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 20d9b891-9c2a-479c-8fd8-93405480dee4. [ 735.240352] nova-conductor[54691]: DEBUG nova.network.neutron [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] [instance: 20d9b891-9c2a-479c-8fd8-93405480dee4] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 735.263906] nova-conductor[54691]: DEBUG nova.network.neutron [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] [instance: 20d9b891-9c2a-479c-8fd8-93405480dee4] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 735.266950] nova-conductor[54691]: DEBUG nova.network.neutron [None req-c7344060-aed2-469b-b21e-91e68c5e9f58 tempest-MigrationsAdminTest-791669259 tempest-MigrationsAdminTest-791669259-project-member] [instance: 20d9b891-9c2a-479c-8fd8-93405480dee4] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 735.320611] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 05bd5de325a146948b461855930946f6 [ 735.687605] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 735.698863] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 735.699299] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 735.699689] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 735.733273] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 735.733714] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 735.734025] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 735.734496] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 735.734810] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 735.735324] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 735.746684] nova-conductor[54692]: DEBUG nova.quota [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Getting quotas for project 204d8f668a894827994c416d74f34c18. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 735.751540] nova-conductor[54692]: DEBUG nova.quota [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Getting quotas for user ea7e09048589421e96a4d19f8e884c8a and project 204d8f668a894827994c416d74f34c18. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 735.765509] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: d866dcf0-c08b-4b12-be83-8de3e69c439f] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 735.765898] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 735.766475] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 735.766475] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 735.772317] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: d866dcf0-c08b-4b12-be83-8de3e69c439f] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 735.772995] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 735.773229] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 735.773632] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 735.790644] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 735.791062] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 735.791454] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 737.303695] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] [instance: 48a0d4f8-4fab-454b-b065-e552647fba3f] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 749951e6-e808-4d96-b4e6-bbcce0326a32, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 48a0d4f8-4fab-454b-b065-e552647fba3f was re-scheduled: Binding failed for port 749951e6-e808-4d96-b4e6-bbcce0326a32, please check neutron logs for more information.\n'] [ 737.303695] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 737.303695] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 48a0d4f8-4fab-454b-b065-e552647fba3f.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 48a0d4f8-4fab-454b-b065-e552647fba3f. [ 737.303695] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] [instance: 48a0d4f8-4fab-454b-b065-e552647fba3f] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 48a0d4f8-4fab-454b-b065-e552647fba3f. [ 737.331474] nova-conductor[54691]: DEBUG nova.network.neutron [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] [instance: 48a0d4f8-4fab-454b-b065-e552647fba3f] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 737.347492] nova-conductor[54691]: DEBUG nova.network.neutron [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] [instance: 48a0d4f8-4fab-454b-b065-e552647fba3f] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 737.351325] nova-conductor[54691]: DEBUG nova.network.neutron [None req-7428bf23-5284-44f0-bc6a-2c44248c37fe tempest-AttachInterfacesUnderV243Test-1451027736 tempest-AttachInterfacesUnderV243Test-1451027736-project-member] [instance: 48a0d4f8-4fab-454b-b065-e552647fba3f] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 745.741337] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell0:SELECT=3 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 745.775678] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=9,nova_cell1:SAVEPOINT=2,nova_cell1:INSERT=23,nova_cell1:RELEASE=2 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 745.784832] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=18,nova_api:DELETE=2 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 745.786058] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=18,nova_api:UPDATE=3,nova_api:DELETE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 745.797089] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Expecting reply to msg acc124b491e14545acff2585dbc5e6af in queue reply_4316ed433d59463c9660e50df524570b [ 745.983213] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg acc124b491e14545acff2585dbc5e6af [ 746.273068] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Expecting reply to msg f409df5e91e3404ea891624c2d9cbbc1 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 746.299566] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 746.324832] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 746.325115] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 746.325318] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 746.363794] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 746.364132] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 746.364337] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 746.364739] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 746.364971] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 746.365115] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 746.375670] nova-conductor[54691]: DEBUG nova.quota [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Getting quotas for project 721d9e362be84a33aa7956177650d445. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 746.376948] nova-conductor[54691]: DEBUG nova.quota [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Getting quotas for user 5b785ade05de40c88f4bf3ef92e5a25f and project 721d9e362be84a33aa7956177650d445. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 746.382739] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: ddcf62db-af40-49d2-9465-66a10f0f2e2d] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 746.383229] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 746.383474] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 746.383816] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 746.388355] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: ddcf62db-af40-49d2-9465-66a10f0f2e2d] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 746.389050] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 746.389285] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 746.389558] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 746.407690] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 746.409304] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 746.409577] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.002s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 746.412897] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg f409df5e91e3404ea891624c2d9cbbc1 [ 746.776398] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 746.792533] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 746.792783] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 746.792950] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 746.800988] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] [instance: 459055aa-ae56-480f-b2c6-e45cc05bb50c] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port ebf4f464-5af2-4630-82e7-8b0520106de4, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 459055aa-ae56-480f-b2c6-e45cc05bb50c was re-scheduled: Binding failed for port ebf4f464-5af2-4630-82e7-8b0520106de4, please check neutron logs for more information.\n'] [ 746.801587] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 746.801835] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 459055aa-ae56-480f-b2c6-e45cc05bb50c.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 459055aa-ae56-480f-b2c6-e45cc05bb50c. [ 746.802163] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] [instance: 459055aa-ae56-480f-b2c6-e45cc05bb50c] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 459055aa-ae56-480f-b2c6-e45cc05bb50c. [ 746.851338] nova-conductor[54692]: DEBUG nova.network.neutron [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] [instance: 459055aa-ae56-480f-b2c6-e45cc05bb50c] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 746.868211] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 746.868485] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 746.868655] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 746.869300] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 746.869564] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 746.869744] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 746.877989] nova-conductor[54692]: DEBUG nova.network.neutron [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] [instance: 459055aa-ae56-480f-b2c6-e45cc05bb50c] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 746.883272] nova-conductor[54692]: DEBUG nova.network.neutron [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] [instance: 459055aa-ae56-480f-b2c6-e45cc05bb50c] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 746.895247] nova-conductor[54692]: DEBUG nova.quota [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Getting quotas for project e049414f9871493398a8e5a07c3ef938. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 746.898886] nova-conductor[54692]: DEBUG nova.quota [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Getting quotas for user 9825cb6e55f8419299e0b4f909b722d6 and project e049414f9871493398a8e5a07c3ef938. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 746.904109] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] [instance: 9f51bbbd-a116-4ebb-b286-4f7db9dbeedb] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 746.904586] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 746.904796] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 746.905013] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 746.909093] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] [instance: 9f51bbbd-a116-4ebb-b286-4f7db9dbeedb] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 746.909746] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 746.909938] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 746.910176] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 746.944440] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 746.944671] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 746.944833] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 751.131145] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] [instance: 5041b2c1-e7f1-4039-b7c4-1f677918025a] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 90717c60-aad7-4e97-b134-e7d76933ad99, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 5041b2c1-e7f1-4039-b7c4-1f677918025a was re-scheduled: Binding failed for port 90717c60-aad7-4e97-b134-e7d76933ad99, please check neutron logs for more information.\n'] [ 751.131664] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 751.131984] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 5041b2c1-e7f1-4039-b7c4-1f677918025a.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 5041b2c1-e7f1-4039-b7c4-1f677918025a. [ 751.132583] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] [instance: 5041b2c1-e7f1-4039-b7c4-1f677918025a] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 5041b2c1-e7f1-4039-b7c4-1f677918025a. [ 751.154314] nova-conductor[54692]: DEBUG nova.network.neutron [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] [instance: 5041b2c1-e7f1-4039-b7c4-1f677918025a] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 751.171171] nova-conductor[54692]: DEBUG nova.network.neutron [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] [instance: 5041b2c1-e7f1-4039-b7c4-1f677918025a] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 751.174314] nova-conductor[54692]: DEBUG nova.network.neutron [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] [instance: 5041b2c1-e7f1-4039-b7c4-1f677918025a] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 752.792325] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=355,nova_cell1:UPDATE=183,nova_cell1:SAVEPOINT=43,nova_cell1:RELEASE=43,nova_cell1:INSERT=21,nova_cell1:DELETE=2 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 753.038731] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] [instance: 896c2790-9107-4d1c-811b-07a87582c0a1] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 5b2db879-2f10-4314-a5f0-c1f237227a8e, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 896c2790-9107-4d1c-811b-07a87582c0a1 was re-scheduled: Binding failed for port 5b2db879-2f10-4314-a5f0-c1f237227a8e, please check neutron logs for more information.\n'] [ 753.038731] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 753.038731] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 896c2790-9107-4d1c-811b-07a87582c0a1.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 896c2790-9107-4d1c-811b-07a87582c0a1. [ 753.038731] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] [instance: 896c2790-9107-4d1c-811b-07a87582c0a1] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 896c2790-9107-4d1c-811b-07a87582c0a1. [ 753.058290] nova-conductor[54691]: DEBUG nova.network.neutron [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] [instance: 896c2790-9107-4d1c-811b-07a87582c0a1] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 753.076291] nova-conductor[54691]: DEBUG nova.network.neutron [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] [instance: 896c2790-9107-4d1c-811b-07a87582c0a1] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 753.081437] nova-conductor[54691]: DEBUG nova.network.neutron [None req-bed3d7b1-e2f6-4237-9326-9c79840a2176 tempest-ListServersNegativeTestJSON-1438981180 tempest-ListServersNegativeTestJSON-1438981180-project-member] [instance: 896c2790-9107-4d1c-811b-07a87582c0a1] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 756.370837] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell0:SELECT=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 756.391994] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:SELECT=3,nova_cell1:SAVEPOINT=1,nova_cell1:INSERT=8,nova_cell1:RELEASE=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 756.400228] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=7,nova_api:UPDATE=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 756.406556] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=5,nova_api:DELETE=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 756.877768] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell0:SELECT=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 756.912467] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=3,nova_cell1:SAVEPOINT=1,nova_cell1:INSERT=8,nova_cell1:RELEASE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 756.930081] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=7 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 756.937376] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=5,nova_api:UPDATE=1,nova_api:DELETE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 757.119341] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:SELECT=377,nova_cell1:SAVEPOINT=36,nova_cell1:RELEASE=36,nova_cell1:UPDATE=184,nova_cell1:INSERT=30 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 762.241657] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] [instance: 1174dcea-6ba6-4a2b-9eb4-85e662691c30] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 28401104-aeb3-4305-8c00-72354272efa2, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 1174dcea-6ba6-4a2b-9eb4-85e662691c30 was re-scheduled: Binding failed for port 28401104-aeb3-4305-8c00-72354272efa2, please check neutron logs for more information.\n'] [ 762.242186] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 762.242508] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1174dcea-6ba6-4a2b-9eb4-85e662691c30.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1174dcea-6ba6-4a2b-9eb4-85e662691c30. [ 762.243079] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] [instance: 1174dcea-6ba6-4a2b-9eb4-85e662691c30] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1174dcea-6ba6-4a2b-9eb4-85e662691c30. [ 762.269020] nova-conductor[54691]: DEBUG nova.network.neutron [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] [instance: 1174dcea-6ba6-4a2b-9eb4-85e662691c30] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 762.286761] nova-conductor[54691]: DEBUG nova.network.neutron [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] [instance: 1174dcea-6ba6-4a2b-9eb4-85e662691c30] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 762.289633] nova-conductor[54691]: DEBUG nova.network.neutron [None req-fdfdaf1a-a896-4bcc-a512-c865e26faa06 tempest-ServerRescueTestJSONUnderV235-1692950365 tempest-ServerRescueTestJSONUnderV235-1692950365-project-member] [instance: 1174dcea-6ba6-4a2b-9eb4-85e662691c30] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 762.456593] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-69a8d882-d36f-4a31-a056-3d9fb89290dd tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Targeting cell 2d4e634d-6292-45c4-9eb1-09b106cacc48(cell1) for conductor method rebuild_instance {{(pid=54692) wrapper /opt/stack/nova/nova/conductor/manager.py:95}} [ 762.456836] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-69a8d882-d36f-4a31-a056-3d9fb89290dd tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 762.457031] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-69a8d882-d36f-4a31-a056-3d9fb89290dd tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 762.457193] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-69a8d882-d36f-4a31-a056-3d9fb89290dd tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 762.464159] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-69a8d882-d36f-4a31-a056-3d9fb89290dd tempest-ServersAdmin275Test-163722317 tempest-ServersAdmin275Test-163722317-project-member] [instance: bbc25649-cfa4-4f5f-ab1e-5f1347780612] No migration record for the rebuild/evacuate request. {{(pid=54692) rebuild_instance /opt/stack/nova/nova/conductor/manager.py:1227}} [ 764.219291] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] [instance: 1a5a4c2e-74e6-4a62-adda-c3d89deecf05] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 701fcdef-8298-4a8d-8739-5a629c9d99e4, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 1a5a4c2e-74e6-4a62-adda-c3d89deecf05 was re-scheduled: Binding failed for port 701fcdef-8298-4a8d-8739-5a629c9d99e4, please check neutron logs for more information.\n'] [ 764.223443] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 764.223443] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1a5a4c2e-74e6-4a62-adda-c3d89deecf05.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1a5a4c2e-74e6-4a62-adda-c3d89deecf05. [ 764.223443] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] [instance: 1a5a4c2e-74e6-4a62-adda-c3d89deecf05] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1a5a4c2e-74e6-4a62-adda-c3d89deecf05. [ 764.245959] nova-conductor[54692]: DEBUG nova.network.neutron [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] [instance: 1a5a4c2e-74e6-4a62-adda-c3d89deecf05] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 764.280890] nova-conductor[54692]: DEBUG nova.network.neutron [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] [instance: 1a5a4c2e-74e6-4a62-adda-c3d89deecf05] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 764.283905] nova-conductor[54692]: DEBUG nova.network.neutron [None req-2ec24e8c-1ccd-40cb-a4ca-85767527de4c tempest-ServerActionsTestOtherA-985572683 tempest-ServerActionsTestOtherA-985572683-project-member] [instance: 1a5a4c2e-74e6-4a62-adda-c3d89deecf05] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 766.115952] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] [instance: 427400bc-f265-40cb-8c1c-ccab7bc94b1c] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port cb3537e6-adcb-4b16-9fc1-bb454d4e1c56, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 427400bc-f265-40cb-8c1c-ccab7bc94b1c was re-scheduled: Binding failed for port cb3537e6-adcb-4b16-9fc1-bb454d4e1c56, please check neutron logs for more information.\n'] [ 766.116689] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 766.116967] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 427400bc-f265-40cb-8c1c-ccab7bc94b1c.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 427400bc-f265-40cb-8c1c-ccab7bc94b1c. [ 766.117243] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] [instance: 427400bc-f265-40cb-8c1c-ccab7bc94b1c] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 427400bc-f265-40cb-8c1c-ccab7bc94b1c. [ 766.145118] nova-conductor[54692]: DEBUG nova.network.neutron [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] [instance: 427400bc-f265-40cb-8c1c-ccab7bc94b1c] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 766.178194] nova-conductor[54692]: DEBUG nova.network.neutron [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] [instance: 427400bc-f265-40cb-8c1c-ccab7bc94b1c] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 766.181744] nova-conductor[54692]: DEBUG nova.network.neutron [None req-c9ea2d0a-832c-4d46-a223-77ca6d1193cf tempest-ServersV294TestFqdnHostnames-1389170272 tempest-ServersV294TestFqdnHostnames-1389170272-project-member] [instance: 427400bc-f265-40cb-8c1c-ccab7bc94b1c] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 770.566388] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: d633de23-23ab-4297-b2d6-2702906b31a9] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port fda4aa7d-b176-4a8a-8311-135c11345fee, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance d633de23-23ab-4297-b2d6-2702906b31a9 was re-scheduled: Binding failed for port fda4aa7d-b176-4a8a-8311-135c11345fee, please check neutron logs for more information.\n'] [ 770.566388] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 770.566743] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d633de23-23ab-4297-b2d6-2702906b31a9.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d633de23-23ab-4297-b2d6-2702906b31a9. [ 770.566865] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: d633de23-23ab-4297-b2d6-2702906b31a9] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d633de23-23ab-4297-b2d6-2702906b31a9. [ 770.599418] nova-conductor[54692]: DEBUG nova.network.neutron [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: d633de23-23ab-4297-b2d6-2702906b31a9] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 770.618027] nova-conductor[54692]: DEBUG nova.network.neutron [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: d633de23-23ab-4297-b2d6-2702906b31a9] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 770.625400] nova-conductor[54692]: DEBUG nova.network.neutron [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: d633de23-23ab-4297-b2d6-2702906b31a9] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 771.730444] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-b9ef691f-78d7-4ed5-a8af-6b87611f17aa tempest-ServersAdmin275Test-881461725 tempest-ServersAdmin275Test-881461725-project-admin] Targeting cell 2d4e634d-6292-45c4-9eb1-09b106cacc48(cell1) for conductor method rebuild_instance {{(pid=54692) wrapper /opt/stack/nova/nova/conductor/manager.py:95}} [ 771.730718] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-b9ef691f-78d7-4ed5-a8af-6b87611f17aa tempest-ServersAdmin275Test-881461725 tempest-ServersAdmin275Test-881461725-project-admin] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 771.733568] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-b9ef691f-78d7-4ed5-a8af-6b87611f17aa tempest-ServersAdmin275Test-881461725 tempest-ServersAdmin275Test-881461725-project-admin] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 771.733568] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-b9ef691f-78d7-4ed5-a8af-6b87611f17aa tempest-ServersAdmin275Test-881461725 tempest-ServersAdmin275Test-881461725-project-admin] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.002s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 771.740145] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-b9ef691f-78d7-4ed5-a8af-6b87611f17aa tempest-ServersAdmin275Test-881461725 tempest-ServersAdmin275Test-881461725-project-admin] [instance: bbc25649-cfa4-4f5f-ab1e-5f1347780612] No migration record for the rebuild/evacuate request. {{(pid=54692) rebuild_instance /opt/stack/nova/nova/conductor/manager.py:1227}} [ 772.467991] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 777.235573] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: 88c5a4ae-ab48-4873-8a42-ecc6d0f99dd0] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 9ad2958c-98f6-40ec-8c0b-17728bf04579, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 88c5a4ae-ab48-4873-8a42-ecc6d0f99dd0 was re-scheduled: Binding failed for port 9ad2958c-98f6-40ec-8c0b-17728bf04579, please check neutron logs for more information.\n'] [ 777.236316] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 777.236562] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 88c5a4ae-ab48-4873-8a42-ecc6d0f99dd0.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 88c5a4ae-ab48-4873-8a42-ecc6d0f99dd0. [ 777.236871] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: 88c5a4ae-ab48-4873-8a42-ecc6d0f99dd0] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 88c5a4ae-ab48-4873-8a42-ecc6d0f99dd0. [ 777.259611] nova-conductor[54692]: DEBUG nova.network.neutron [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: 88c5a4ae-ab48-4873-8a42-ecc6d0f99dd0] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 777.278066] nova-conductor[54692]: DEBUG nova.network.neutron [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: 88c5a4ae-ab48-4873-8a42-ecc6d0f99dd0] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 777.281417] nova-conductor[54692]: DEBUG nova.network.neutron [None req-8b33de38-de92-45c0-80f7-7424439a9c46 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: 88c5a4ae-ab48-4873-8a42-ecc6d0f99dd0] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 779.225628] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Expecting reply to msg e3995a3555d342a78619a6c7743d0b2d in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 779.487622] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg e3995a3555d342a78619a6c7743d0b2d [ 779.733840] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Took 0.51 seconds to select destinations for 2 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 779.749425] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 779.749764] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 779.750023] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 779.777261] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 779.777617] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 779.777852] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 779.798400] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 779.798720] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 779.798955] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 779.799404] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 779.799653] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 779.799869] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 779.807515] nova-conductor[54692]: DEBUG nova.quota [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Getting quotas for project 6a34141edbd4482d9f709ad211179a90. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 779.811090] nova-conductor[54692]: DEBUG nova.quota [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Getting quotas for user ce8f935dca594cfa87b680e92f1c492c and project 6a34141edbd4482d9f709ad211179a90. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 779.816085] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: 1f0cc9c9-0d9f-40ea-a2cf-ff94c793ff4e] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 779.816592] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 779.816863] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 779.817133] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 779.822152] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: 1f0cc9c9-0d9f-40ea-a2cf-ff94c793ff4e] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 779.822887] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 779.823171] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 779.823401] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 779.838711] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 779.839026] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 779.839267] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 779.845308] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: 1f90a84e-6c32-4846-8908-128323e33e98] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 779.845869] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 779.846117] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 779.846344] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 779.848928] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: 1f90a84e-6c32-4846-8908-128323e33e98] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 779.849686] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 779.849936] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 779.850154] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 779.861756] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 779.862022] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 779.862255] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 789.805008] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell0:SELECT=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 789.852896] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=13,nova_cell1:UPDATE=6,nova_cell1:INSERT=16 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 789.857747] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=11,nova_api:UPDATE=1,nova_api:DELETE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 789.859721] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=8,nova_api:UPDATE=1,nova_api:DELETE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 792.051224] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] [instance: 6a7e7ba2-a972-48bd-83b7-8d1b6a3289ca] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 21f35458-40bc-40fc-8381-9814ec25b9cb, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 6a7e7ba2-a972-48bd-83b7-8d1b6a3289ca was re-scheduled: Binding failed for port 21f35458-40bc-40fc-8381-9814ec25b9cb, please check neutron logs for more information.\n'] [ 792.051802] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 792.052065] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 6a7e7ba2-a972-48bd-83b7-8d1b6a3289ca.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 6a7e7ba2-a972-48bd-83b7-8d1b6a3289ca. [ 792.052410] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] [instance: 6a7e7ba2-a972-48bd-83b7-8d1b6a3289ca] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 6a7e7ba2-a972-48bd-83b7-8d1b6a3289ca. [ 792.082662] nova-conductor[54692]: DEBUG nova.network.neutron [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] [instance: 6a7e7ba2-a972-48bd-83b7-8d1b6a3289ca] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 792.104038] nova-conductor[54692]: DEBUG nova.network.neutron [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] [instance: 6a7e7ba2-a972-48bd-83b7-8d1b6a3289ca] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 792.106859] nova-conductor[54692]: DEBUG nova.network.neutron [None req-7744b891-7149-4f81-a9d6-800f73ed441e tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] [instance: 6a7e7ba2-a972-48bd-83b7-8d1b6a3289ca] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 795.245994] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Expecting reply to msg 33d6aa76f6774b38ae3df328458d83de in queue reply_4316ed433d59463c9660e50df524570b [ 795.386492] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 33d6aa76f6774b38ae3df328458d83de [ 795.748095] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 795.760789] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 795.761014] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 795.761184] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 795.788471] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 795.788705] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 795.788891] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 795.789253] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 795.789475] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 795.789663] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 795.798008] nova-conductor[54691]: DEBUG nova.quota [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Getting quotas for project 7c343033abe74ef28f09e614ffad9254. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 795.800059] nova-conductor[54691]: DEBUG nova.quota [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Getting quotas for user 613209c7c103464ea473f32bcf0116b1 and project 7c343033abe74ef28f09e614ffad9254. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 795.807234] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] [instance: e2a1c5a7-34a1-4520-9b37-70f6087f0314] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 795.807673] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 795.807888] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 795.808060] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 795.812961] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] [instance: e2a1c5a7-34a1-4520-9b37-70f6087f0314] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 795.813593] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 795.813788] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 795.814083] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 795.837032] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 795.837032] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 795.837032] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 801.044090] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] [instance: 3135faf8-b918-42f5-bcaa-c23a19166bb3] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 7546ffee-78bf-462a-a94a-d087f5f1b40d, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 3135faf8-b918-42f5-bcaa-c23a19166bb3 was re-scheduled: Binding failed for port 7546ffee-78bf-462a-a94a-d087f5f1b40d, please check neutron logs for more information.\n'] [ 801.044762] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 801.044999] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 3135faf8-b918-42f5-bcaa-c23a19166bb3.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 3135faf8-b918-42f5-bcaa-c23a19166bb3. [ 801.045306] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] [instance: 3135faf8-b918-42f5-bcaa-c23a19166bb3] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 3135faf8-b918-42f5-bcaa-c23a19166bb3. [ 801.066109] nova-conductor[54691]: DEBUG nova.network.neutron [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] [instance: 3135faf8-b918-42f5-bcaa-c23a19166bb3] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 801.083604] nova-conductor[54691]: DEBUG nova.network.neutron [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] [instance: 3135faf8-b918-42f5-bcaa-c23a19166bb3] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 801.086325] nova-conductor[54691]: DEBUG nova.network.neutron [None req-ea32325d-dfbc-429e-8784-2a6cf8c1f4da tempest-ServersNegativeTestMultiTenantJSON-2037149078 tempest-ServersNegativeTestMultiTenantJSON-2037149078-project-member] [instance: 3135faf8-b918-42f5-bcaa-c23a19166bb3] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 805.794993] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell0:SELECT=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 805.815991] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:SELECT=3,nova_cell1:INSERT=7 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 805.829926] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=5,nova_api:UPDATE=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 805.831504] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=7,nova_api:DELETE=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 806.238109] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] [instance: 97927074-a676-4ebd-aa7a-fe92e3ce1be6] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 95eb6170-9245-415c-9bdb-5a5437c7ddb8, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 97927074-a676-4ebd-aa7a-fe92e3ce1be6 was re-scheduled: Binding failed for port 95eb6170-9245-415c-9bdb-5a5437c7ddb8, please check neutron logs for more information.\n'] [ 806.238654] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 806.238893] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 97927074-a676-4ebd-aa7a-fe92e3ce1be6.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 97927074-a676-4ebd-aa7a-fe92e3ce1be6. [ 806.239165] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] [instance: 97927074-a676-4ebd-aa7a-fe92e3ce1be6] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 97927074-a676-4ebd-aa7a-fe92e3ce1be6. [ 806.268904] nova-conductor[54692]: DEBUG nova.network.neutron [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] [instance: 97927074-a676-4ebd-aa7a-fe92e3ce1be6] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 806.284380] nova-conductor[54692]: DEBUG nova.network.neutron [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] [instance: 97927074-a676-4ebd-aa7a-fe92e3ce1be6] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 806.287100] nova-conductor[54692]: DEBUG nova.network.neutron [None req-9640061f-8bc3-474f-9486-8a68eaa19a56 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] [instance: 97927074-a676-4ebd-aa7a-fe92e3ce1be6] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 808.062398] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] [instance: 61f31838-2fd1-4804-82b7-56b3a08c3ff5] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 993db6d2-a560-4b87-9cd6-1bd027525b39, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 61f31838-2fd1-4804-82b7-56b3a08c3ff5 was re-scheduled: Binding failed for port 993db6d2-a560-4b87-9cd6-1bd027525b39, please check neutron logs for more information.\n'] [ 808.062980] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 808.063212] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 61f31838-2fd1-4804-82b7-56b3a08c3ff5.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 61f31838-2fd1-4804-82b7-56b3a08c3ff5. [ 808.063423] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] [instance: 61f31838-2fd1-4804-82b7-56b3a08c3ff5] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 61f31838-2fd1-4804-82b7-56b3a08c3ff5. [ 808.088814] nova-conductor[54691]: DEBUG nova.network.neutron [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] [instance: 61f31838-2fd1-4804-82b7-56b3a08c3ff5] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 808.103123] nova-conductor[54691]: DEBUG nova.network.neutron [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] [instance: 61f31838-2fd1-4804-82b7-56b3a08c3ff5] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 808.105927] nova-conductor[54691]: DEBUG nova.network.neutron [None req-e1534752-5daa-4e5a-9b8b-021679612a65 tempest-ListImageFiltersTestJSON-1393988037 tempest-ListImageFiltersTestJSON-1393988037-project-member] [instance: 61f31838-2fd1-4804-82b7-56b3a08c3ff5] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 813.053767] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Expecting reply to msg 37f184bf39a24dc49c55764d10c09725 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 813.220334] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 37f184bf39a24dc49c55764d10c09725 [ 813.325331] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=321,nova_cell1:UPDATE=151,nova_cell1:INSERT=16,nova_cell1:SAVEPOINT=32,nova_cell1:RELEASE=32,nova_cell1:DELETE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 813.555911] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 813.568463] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 813.568695] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 813.568859] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 813.604059] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 813.604297] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 813.604465] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 813.604831] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 813.605023] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 813.605181] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 813.629005] nova-conductor[54692]: DEBUG nova.quota [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Getting quotas for project 17dd5c4ec5e54de8867ffaa08754b7cd. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 813.633127] nova-conductor[54692]: DEBUG nova.quota [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Getting quotas for user d68b7999531b4acb88e76d4393cee695 and project 17dd5c4ec5e54de8867ffaa08754b7cd. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 813.642858] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] [instance: 7d7cc825-7dcc-4b21-bf4c-886de451863a] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 813.642858] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 813.642858] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 813.642858] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 813.649781] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] [instance: 7d7cc825-7dcc-4b21-bf4c-886de451863a] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 813.649781] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 813.649781] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 813.649781] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 813.683083] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 813.683565] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 813.684158] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 814.853729] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Expecting reply to msg e9d0e907497947ab83c6713f8204054d in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 815.024677] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg e9d0e907497947ab83c6713f8204054d [ 815.360633] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: e7b5d223-a1c7-4b17-a14b-9d75deeb3ac0] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 1a7e7d7a-929e-4293-87d4-20021d6f1e74, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance e7b5d223-a1c7-4b17-a14b-9d75deeb3ac0 was re-scheduled: Binding failed for port 1a7e7d7a-929e-4293-87d4-20021d6f1e74, please check neutron logs for more information.\n'] [ 815.361741] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 815.361948] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 815.362197] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance e7b5d223-a1c7-4b17-a14b-9d75deeb3ac0.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance e7b5d223-a1c7-4b17-a14b-9d75deeb3ac0. [ 815.362405] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: e7b5d223-a1c7-4b17-a14b-9d75deeb3ac0] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance e7b5d223-a1c7-4b17-a14b-9d75deeb3ac0. [ 815.395638] nova-conductor[54691]: DEBUG nova.network.neutron [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: e7b5d223-a1c7-4b17-a14b-9d75deeb3ac0] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 815.398261] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 815.399290] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 815.399290] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 815.415149] nova-conductor[54691]: DEBUG nova.network.neutron [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: e7b5d223-a1c7-4b17-a14b-9d75deeb3ac0] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 815.416876] nova-conductor[54691]: DEBUG nova.network.neutron [None req-6652971a-9d51-41c0-9fb9-f9a8cf53e955 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: e7b5d223-a1c7-4b17-a14b-9d75deeb3ac0] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 815.434698] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 815.434944] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 815.435110] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 815.435469] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 815.435649] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 815.435807] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 815.445837] nova-conductor[54692]: DEBUG nova.quota [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Getting quotas for project c41cd9d2bbc4443db64facd9d3f956d1. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 815.449233] nova-conductor[54692]: DEBUG nova.quota [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Getting quotas for user 52362e53439340569e09b18293dd5a3a and project c41cd9d2bbc4443db64facd9d3f956d1. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 815.455332] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] [instance: 86be55c9-82d5-4a34-b628-4729c42d83c1] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 815.455876] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 815.456115] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 815.456280] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 815.462048] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] [instance: 86be55c9-82d5-4a34-b628-4729c42d83c1] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 815.462622] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 815.462894] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 815.463079] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 815.484068] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 815.484360] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 815.484530] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 816.352160] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Expecting reply to msg 49f0202c96c14abfb0541881301bb2a6 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 816.492759] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 49f0202c96c14abfb0541881301bb2a6 [ 816.854632] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 816.867556] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 816.867777] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 816.867936] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 816.903245] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 816.903427] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 816.903600] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 816.903955] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 816.904155] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 816.904312] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 816.912920] nova-conductor[54692]: DEBUG nova.quota [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Getting quotas for project de11ac365f704c2986c782a5918f802f. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 816.915861] nova-conductor[54692]: DEBUG nova.quota [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Getting quotas for user c43c33165931465494c0a7da171126e4 and project de11ac365f704c2986c782a5918f802f. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 816.920836] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: 53625c0d-203a-45f0-b899-5d2673413a53] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 816.921289] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 816.921486] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 816.921652] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 816.925184] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: 53625c0d-203a-45f0-b899-5d2673413a53] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 816.925801] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 816.925999] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 816.926160] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 816.939700] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 816.939940] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 816.940138] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 817.550966] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:SELECT=347,nova_cell1:UPDATE=183,nova_cell1:INSERT=32,nova_cell1:SAVEPOINT=34,nova_cell1:RELEASE=34,nova_cell1:DELETE=2 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 819.408937] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] [instance: fd2547e6-9449-4839-8ebb-f02f01a85b8c] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port e0c0a849-c8ce-4d35-a085-a6276922400b, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance fd2547e6-9449-4839-8ebb-f02f01a85b8c was re-scheduled: Binding failed for port e0c0a849-c8ce-4d35-a085-a6276922400b, please check neutron logs for more information.\n'] [ 819.409586] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 819.409825] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance fd2547e6-9449-4839-8ebb-f02f01a85b8c.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance fd2547e6-9449-4839-8ebb-f02f01a85b8c. [ 819.410131] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] [instance: fd2547e6-9449-4839-8ebb-f02f01a85b8c] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance fd2547e6-9449-4839-8ebb-f02f01a85b8c. [ 819.449158] nova-conductor[54692]: DEBUG nova.network.neutron [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] [instance: fd2547e6-9449-4839-8ebb-f02f01a85b8c] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 819.470841] nova-conductor[54692]: DEBUG nova.network.neutron [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] [instance: fd2547e6-9449-4839-8ebb-f02f01a85b8c] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 819.471476] nova-conductor[54692]: DEBUG nova.network.neutron [None req-977306e0-8d53-44bd-9f97-fbb2d1b9be52 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] [instance: fd2547e6-9449-4839-8ebb-f02f01a85b8c] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 821.261785] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] [instance: 5014a870-32fd-4166-9446-c41ed2a4954e] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 30f94bf9-c081-4c9e-8c94-4a50ed6b7af0, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 5014a870-32fd-4166-9446-c41ed2a4954e was re-scheduled: Binding failed for port 30f94bf9-c081-4c9e-8c94-4a50ed6b7af0, please check neutron logs for more information.\n'] [ 821.262541] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 821.262810] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 5014a870-32fd-4166-9446-c41ed2a4954e.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 5014a870-32fd-4166-9446-c41ed2a4954e. [ 821.263019] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] [instance: 5014a870-32fd-4166-9446-c41ed2a4954e] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 5014a870-32fd-4166-9446-c41ed2a4954e. [ 821.291284] nova-conductor[54691]: DEBUG nova.network.neutron [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] [instance: 5014a870-32fd-4166-9446-c41ed2a4954e] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 821.312455] nova-conductor[54691]: DEBUG nova.network.neutron [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] [instance: 5014a870-32fd-4166-9446-c41ed2a4954e] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 821.315869] nova-conductor[54691]: DEBUG nova.network.neutron [None req-383ce570-73cb-447d-a3b6-a6192d1a0d22 tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] [instance: 5014a870-32fd-4166-9446-c41ed2a4954e] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 822.350138] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Expecting reply to msg 54db6010794740e683ac28fa3e1230eb in queue reply_4316ed433d59463c9660e50df524570b [ 822.493665] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 54db6010794740e683ac28fa3e1230eb [ 822.740135] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Expecting reply to msg 137f274ab2d346dba665d8a3064bf7eb in queue reply_4316ed433d59463c9660e50df524570b [ 822.852644] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 822.865800] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 822.866076] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 822.866196] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 822.868641] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 137f274ab2d346dba665d8a3064bf7eb [ 822.898676] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 822.898903] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 822.899063] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 822.899447] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 822.899630] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 822.899783] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 822.911214] nova-conductor[54691]: DEBUG nova.quota [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Getting quotas for project 9a349372836346d8983c95b50d980e37. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 822.914437] nova-conductor[54691]: DEBUG nova.quota [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Getting quotas for user 23c687bad35b409ab96bfd7cde54fd27 and project 9a349372836346d8983c95b50d980e37. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 822.921929] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] [instance: f7ed0a84-a3ca-46f8-8cfb-da2311c9cb90] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 822.922595] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 822.923054] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 822.923292] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 822.930096] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] [instance: f7ed0a84-a3ca-46f8-8cfb-da2311c9cb90] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 822.930096] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 822.930096] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 822.930419] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 822.955388] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 822.955603] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 822.955764] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 823.098054] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] [instance: b49cde53-0ef7-44a4-b894-e37d1cbf41a7] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port cc0c8455-7089-49be-8598-cb883d892f6d, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance b49cde53-0ef7-44a4-b894-e37d1cbf41a7 was re-scheduled: Binding failed for port cc0c8455-7089-49be-8598-cb883d892f6d, please check neutron logs for more information.\n'] [ 823.099674] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 823.099674] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance b49cde53-0ef7-44a4-b894-e37d1cbf41a7.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance b49cde53-0ef7-44a4-b894-e37d1cbf41a7. [ 823.099674] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] [instance: b49cde53-0ef7-44a4-b894-e37d1cbf41a7] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance b49cde53-0ef7-44a4-b894-e37d1cbf41a7. [ 823.123859] nova-conductor[54691]: DEBUG nova.network.neutron [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] [instance: b49cde53-0ef7-44a4-b894-e37d1cbf41a7] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 823.139505] nova-conductor[54691]: DEBUG nova.network.neutron [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] [instance: b49cde53-0ef7-44a4-b894-e37d1cbf41a7] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 823.146549] nova-conductor[54691]: DEBUG nova.network.neutron [None req-eabfdd60-e769-4bf8-86c2-89886d2aa6f5 tempest-InstanceActionsNegativeTestJSON-21200735 tempest-InstanceActionsNegativeTestJSON-21200735-project-member] [instance: b49cde53-0ef7-44a4-b894-e37d1cbf41a7] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 823.242344] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 823.253665] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 823.253888] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 823.254051] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 823.274117] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 823.274332] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 823.274497] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 823.274857] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 823.275038] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 823.275193] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 823.282460] nova-conductor[54691]: DEBUG nova.quota [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Getting quotas for project 8003d16b1cc048bc94407d028838a4f6. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 823.284411] nova-conductor[54691]: DEBUG nova.quota [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Getting quotas for user cd65b3ab163145e1bf370aa0dc63f65c and project 8003d16b1cc048bc94407d028838a4f6. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 823.289131] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] [instance: 570ee4e0-caeb-41f9-82d2-fa009f23581d] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 823.289550] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 823.289747] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 823.289917] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 823.293216] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] [instance: 570ee4e0-caeb-41f9-82d2-fa009f23581d] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 823.300218] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 823.300218] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 823.300218] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 823.305687] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 823.305879] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 823.306037] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 826.907657] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell0:SELECT=3 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 826.928471] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=9,nova_cell1:SAVEPOINT=2,nova_cell1:INSERT=23,nova_cell1:RELEASE=2 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 826.936401] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=18,nova_api:UPDATE=1,nova_api:DELETE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 826.938704] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=18,nova_api:DELETE=2,nova_api:UPDATE=2 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 827.675213] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] [instance: 25e906f6-ec98-4c99-83c7-43f94c5790dd] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 77e7b973-54f1-4090-9007-9bdb448043dd, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 25e906f6-ec98-4c99-83c7-43f94c5790dd was re-scheduled: Binding failed for port 77e7b973-54f1-4090-9007-9bdb448043dd, please check neutron logs for more information.\n'] [ 827.675823] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 827.676062] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 25e906f6-ec98-4c99-83c7-43f94c5790dd.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 25e906f6-ec98-4c99-83c7-43f94c5790dd. [ 827.676365] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] [instance: 25e906f6-ec98-4c99-83c7-43f94c5790dd] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 25e906f6-ec98-4c99-83c7-43f94c5790dd. [ 827.703057] nova-conductor[54691]: DEBUG nova.network.neutron [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] [instance: 25e906f6-ec98-4c99-83c7-43f94c5790dd] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 827.724275] nova-conductor[54691]: DEBUG nova.network.neutron [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] [instance: 25e906f6-ec98-4c99-83c7-43f94c5790dd] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 827.727596] nova-conductor[54691]: DEBUG nova.network.neutron [None req-f3bb6349-5815-4d43-914d-567184e1cac0 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] [instance: 25e906f6-ec98-4c99-83c7-43f94c5790dd] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 828.642786] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Expecting reply to msg ab201c19474f4d049440b17fe3860476 in queue reply_4316ed433d59463c9660e50df524570b [ 828.871739] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg ab201c19474f4d049440b17fe3860476 [ 829.146172] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 829.159674] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 829.159897] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 829.160078] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 829.192415] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 829.192415] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 829.192415] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 829.192530] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 829.192691] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 829.192850] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 829.228734] nova-conductor[54691]: DEBUG nova.quota [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Getting quotas for project 06405d5d1bdb4917a235c5602d815dc5. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 829.236917] nova-conductor[54691]: DEBUG nova.quota [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Getting quotas for user 52b93ec242174e5ab53c6b97f5396c6f and project 06405d5d1bdb4917a235c5602d815dc5. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 829.242933] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] [instance: 7d047648-facb-4d58-acf7-a99a3f7adf65] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 829.243394] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 829.243604] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 829.243994] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 829.249075] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] [instance: 7d047648-facb-4d58-acf7-a99a3f7adf65] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 829.249751] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 829.249961] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 829.250129] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 829.265248] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 829.265481] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 829.265659] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 829.861715] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Expecting reply to msg b3059e7c8b3e46df87a204f9a0024e71 in queue reply_4316ed433d59463c9660e50df524570b [ 830.007816] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg b3059e7c8b3e46df87a204f9a0024e71 [ 830.364656] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 830.376422] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 830.376653] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 830.376894] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 830.402280] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 830.402498] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 830.402663] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 830.403024] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 830.403205] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 830.403359] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 830.411707] nova-conductor[54691]: DEBUG nova.quota [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Getting quotas for project d781d0a045444391bc070520be1b6431. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 830.414659] nova-conductor[54691]: DEBUG nova.quota [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Getting quotas for user 6a6d2a917fec48e48e943137dff043fd and project d781d0a045444391bc070520be1b6431. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 830.419404] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] [instance: fa27cfe2-6f7f-4397-bcc5-147d6865f404] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 830.419829] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 830.420040] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 830.420208] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 830.424464] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] [instance: fa27cfe2-6f7f-4397-bcc5-147d6865f404] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 830.425059] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 830.425248] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 830.425407] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 830.438303] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 830.438502] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 830.438713] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 834.256612] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] [instance: 99735498-3c85-4a01-bc05-4def014618a8] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 6696136d-efd5-48c3-84b6-c8fcd9d3aa8a, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 99735498-3c85-4a01-bc05-4def014618a8 was re-scheduled: Binding failed for port 6696136d-efd5-48c3-84b6-c8fcd9d3aa8a, please check neutron logs for more information.\n'] [ 834.257246] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 834.257521] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 99735498-3c85-4a01-bc05-4def014618a8.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 99735498-3c85-4a01-bc05-4def014618a8. [ 834.257833] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] [instance: 99735498-3c85-4a01-bc05-4def014618a8] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 99735498-3c85-4a01-bc05-4def014618a8. [ 834.285652] nova-conductor[54691]: DEBUG nova.network.neutron [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] [instance: 99735498-3c85-4a01-bc05-4def014618a8] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 834.312279] nova-conductor[54691]: DEBUG nova.network.neutron [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] [instance: 99735498-3c85-4a01-bc05-4def014618a8] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 834.313627] nova-conductor[54691]: DEBUG nova.network.neutron [None req-90a7f354-b213-4e50-b09e-e7191df4e099 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] [instance: 99735498-3c85-4a01-bc05-4def014618a8] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 835.792175] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-346a9f43-e251-431e-ab08-cde60afc8b34 tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Targeting cell 2d4e634d-6292-45c4-9eb1-09b106cacc48(cell1) for conductor method rebuild_instance {{(pid=54692) wrapper /opt/stack/nova/nova/conductor/manager.py:95}} [ 835.792508] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-346a9f43-e251-431e-ab08-cde60afc8b34 tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 835.792777] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-346a9f43-e251-431e-ab08-cde60afc8b34 tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 835.792966] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-346a9f43-e251-431e-ab08-cde60afc8b34 tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 835.804395] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-346a9f43-e251-431e-ab08-cde60afc8b34 tempest-ServerShowV257Test-1223842529 tempest-ServerShowV257Test-1223842529-project-member] [instance: 5d2465e7-e473-46e9-a8ce-cc58dcdbd28f] No migration record for the rebuild/evacuate request. {{(pid=54692) rebuild_instance /opt/stack/nova/nova/conductor/manager.py:1227}} [ 838.504495] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] [instance: 6c4ac72f-1636-4f4c-928c-0a6fe895ce37] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 902fe20e-4e9c-49b8-b7b0-c5b27cf5de23, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 6c4ac72f-1636-4f4c-928c-0a6fe895ce37 was re-scheduled: Binding failed for port 902fe20e-4e9c-49b8-b7b0-c5b27cf5de23, please check neutron logs for more information.\n'] [ 838.505117] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 838.505353] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 6c4ac72f-1636-4f4c-928c-0a6fe895ce37.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 6c4ac72f-1636-4f4c-928c-0a6fe895ce37. [ 838.505664] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] [instance: 6c4ac72f-1636-4f4c-928c-0a6fe895ce37] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 6c4ac72f-1636-4f4c-928c-0a6fe895ce37. [ 838.535975] nova-conductor[54692]: DEBUG nova.network.neutron [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] [instance: 6c4ac72f-1636-4f4c-928c-0a6fe895ce37] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 838.603438] nova-conductor[54692]: DEBUG nova.network.neutron [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] [instance: 6c4ac72f-1636-4f4c-928c-0a6fe895ce37] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 838.606640] nova-conductor[54692]: DEBUG nova.network.neutron [None req-3394e696-4294-404c-886b-56dade178294 tempest-ServerRescueNegativeTestJSON-107795642 tempest-ServerRescueNegativeTestJSON-107795642-project-member] [instance: 6c4ac72f-1636-4f4c-928c-0a6fe895ce37] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 840.408498] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell0:SELECT=4 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 840.427296] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:SELECT=12,nova_cell1:INSERT=29,nova_cell1:SAVEPOINT=1,nova_cell1:RELEASE=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 840.433915] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=26,nova_api:UPDATE=1,nova_api:DELETE=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 840.436605] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=22,nova_api:UPDATE=3,nova_api:DELETE=3 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 845.791737] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 845.808472] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 845.823015] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=4,nova_cell1:UPDATE=3,nova_cell1:INSERT=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 847.339359] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] [instance: 451b9e70-f202-4a10-8b19-34b6167ef14d] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 6b2abb82-a5d2-4157-8b93-2ef7bfb32a0b, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 451b9e70-f202-4a10-8b19-34b6167ef14d was re-scheduled: Binding failed for port 6b2abb82-a5d2-4157-8b93-2ef7bfb32a0b, please check neutron logs for more information.\n'] [ 847.341601] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 847.341881] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 451b9e70-f202-4a10-8b19-34b6167ef14d.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 451b9e70-f202-4a10-8b19-34b6167ef14d. [ 847.342110] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] [instance: 451b9e70-f202-4a10-8b19-34b6167ef14d] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 451b9e70-f202-4a10-8b19-34b6167ef14d. [ 847.362877] nova-conductor[54692]: DEBUG nova.network.neutron [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] [instance: 451b9e70-f202-4a10-8b19-34b6167ef14d] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 847.377308] nova-conductor[54692]: DEBUG nova.network.neutron [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] [instance: 451b9e70-f202-4a10-8b19-34b6167ef14d] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 847.381343] nova-conductor[54692]: DEBUG nova.network.neutron [None req-c3fe70d3-6af1-4cb6-aa74-8d5614569729 tempest-SecurityGroupsTestJSON-257580290 tempest-SecurityGroupsTestJSON-257580290-project-member] [instance: 451b9e70-f202-4a10-8b19-34b6167ef14d] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 851.590747] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] [instance: ee85f964-db45-404c-9016-b618748ec63c] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 33c97501-12cc-4d90-b4eb-9d442da54929, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance ee85f964-db45-404c-9016-b618748ec63c was re-scheduled: Binding failed for port 33c97501-12cc-4d90-b4eb-9d442da54929, please check neutron logs for more information.\n'] [ 851.591314] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 851.591543] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ee85f964-db45-404c-9016-b618748ec63c.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ee85f964-db45-404c-9016-b618748ec63c. [ 851.591753] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] [instance: ee85f964-db45-404c-9016-b618748ec63c] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ee85f964-db45-404c-9016-b618748ec63c. [ 851.612395] nova-conductor[54692]: DEBUG nova.network.neutron [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] [instance: ee85f964-db45-404c-9016-b618748ec63c] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 851.637165] nova-conductor[54692]: DEBUG nova.network.neutron [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] [instance: ee85f964-db45-404c-9016-b618748ec63c] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 851.647802] nova-conductor[54692]: DEBUG nova.network.neutron [None req-a2af6359-9dfe-4f11-bc59-a146ba7d4955 tempest-ServerMetadataTestJSON-1364403100 tempest-ServerMetadataTestJSON-1364403100-project-member] [instance: ee85f964-db45-404c-9016-b618748ec63c] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 856.112101] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 856.112101] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 856.112101] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 856.135689] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Expecting reply to msg cab65510788c4f0ebc3fcdd8e18b83e5 in queue reply_4316ed433d59463c9660e50df524570b [ 856.268613] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg cab65510788c4f0ebc3fcdd8e18b83e5 [ 856.638493] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 856.650296] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 856.650521] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 856.650752] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 856.685181] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 856.685539] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 856.685713] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 856.686076] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 856.686250] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 856.686400] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 856.695361] nova-conductor[54691]: DEBUG nova.quota [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Getting quotas for project cc9bb22c5b864296a51086af6ddd40d7. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 856.698116] nova-conductor[54691]: DEBUG nova.quota [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Getting quotas for user cf79d14f7faf42139e1e8f342e1fe9b6 and project cc9bb22c5b864296a51086af6ddd40d7. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 856.702826] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] [instance: 768cf3b2-227f-463a-b192-79c50874cded] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 856.703222] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 856.703412] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 856.703574] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 856.709551] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] [instance: 768cf3b2-227f-463a-b192-79c50874cded] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 856.710169] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 856.710359] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 856.710522] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 856.724724] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 856.724962] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 856.725224] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 858.017847] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: d7947374-0e8f-4d00-a430-923f0f2bb858] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port b4e7353c-b6e4-4257-a9f3-5b3989087fad, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance d7947374-0e8f-4d00-a430-923f0f2bb858 was re-scheduled: Binding failed for port b4e7353c-b6e4-4257-a9f3-5b3989087fad, please check neutron logs for more information.\n'] [ 858.017847] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 858.017847] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d7947374-0e8f-4d00-a430-923f0f2bb858.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d7947374-0e8f-4d00-a430-923f0f2bb858. [ 858.017847] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: d7947374-0e8f-4d00-a430-923f0f2bb858] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d7947374-0e8f-4d00-a430-923f0f2bb858. [ 858.037673] nova-conductor[54691]: DEBUG nova.network.neutron [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: d7947374-0e8f-4d00-a430-923f0f2bb858] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 858.174592] nova-conductor[54691]: DEBUG nova.network.neutron [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: d7947374-0e8f-4d00-a430-923f0f2bb858] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 858.188285] nova-conductor[54691]: DEBUG nova.network.neutron [None req-3d4299ab-4e1b-46b6-a286-83dfe570cd7f tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: d7947374-0e8f-4d00-a430-923f0f2bb858] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 859.529560] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Expecting reply to msg 7f022c7747524903ac908b66060f462b in queue reply_4316ed433d59463c9660e50df524570b [ 859.682390] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 7f022c7747524903ac908b66060f462b [ 860.031849] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 860.045591] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 860.045874] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 860.045986] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 860.075721] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 860.075938] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 860.076135] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 860.076492] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 860.076671] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 860.076828] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 860.085400] nova-conductor[54691]: DEBUG nova.quota [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Getting quotas for project adf774b7c36f46219e7dfdec663fe624. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 860.088587] nova-conductor[54691]: DEBUG nova.quota [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Getting quotas for user 949bec663bc44ef3b1d3eb8c8f6912a9 and project adf774b7c36f46219e7dfdec663fe624. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 860.097705] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 1f1dd01d-006d-4569-8684-e1eb913ba5d4] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 860.098197] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 860.098398] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 860.098556] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 860.102819] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 1f1dd01d-006d-4569-8684-e1eb913ba5d4] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 860.103431] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 860.103620] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 860.103775] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 860.127387] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 860.127612] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 860.127773] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 862.156270] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] [instance: 76c79f9e-7ce6-4af9-b937-8e4bd32d39fe] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port f6facc53-17f2-461e-880c-00ea6c9b33c0, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 76c79f9e-7ce6-4af9-b937-8e4bd32d39fe was re-scheduled: Binding failed for port f6facc53-17f2-461e-880c-00ea6c9b33c0, please check neutron logs for more information.\n'] [ 862.156829] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 862.157069] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 76c79f9e-7ce6-4af9-b937-8e4bd32d39fe.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 76c79f9e-7ce6-4af9-b937-8e4bd32d39fe. [ 862.157271] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] [instance: 76c79f9e-7ce6-4af9-b937-8e4bd32d39fe] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 76c79f9e-7ce6-4af9-b937-8e4bd32d39fe. [ 862.176275] nova-conductor[54692]: DEBUG nova.network.neutron [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] [instance: 76c79f9e-7ce6-4af9-b937-8e4bd32d39fe] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 862.213163] nova-conductor[54692]: DEBUG nova.network.neutron [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] [instance: 76c79f9e-7ce6-4af9-b937-8e4bd32d39fe] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 862.216286] nova-conductor[54692]: DEBUG nova.network.neutron [None req-f5be47be-422f-4152-8cb3-4890719d2ec6 tempest-ImagesOneServerNegativeTestJSON-1164545313 tempest-ImagesOneServerNegativeTestJSON-1164545313-project-member] [instance: 76c79f9e-7ce6-4af9-b937-8e4bd32d39fe] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 864.021261] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: d866dcf0-c08b-4b12-be83-8de3e69c439f] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 0ce0a4e3-87d1-48d5-bd07-a94ee27198e8, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance d866dcf0-c08b-4b12-be83-8de3e69c439f was re-scheduled: Binding failed for port 0ce0a4e3-87d1-48d5-bd07-a94ee27198e8, please check neutron logs for more information.\n'] [ 864.021957] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 864.022252] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d866dcf0-c08b-4b12-be83-8de3e69c439f.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d866dcf0-c08b-4b12-be83-8de3e69c439f. [ 864.022518] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: d866dcf0-c08b-4b12-be83-8de3e69c439f] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d866dcf0-c08b-4b12-be83-8de3e69c439f. [ 864.053928] nova-conductor[54692]: DEBUG nova.network.neutron [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: d866dcf0-c08b-4b12-be83-8de3e69c439f] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 864.067737] nova-conductor[54692]: DEBUG nova.network.neutron [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: d866dcf0-c08b-4b12-be83-8de3e69c439f] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 864.072208] nova-conductor[54692]: DEBUG nova.network.neutron [None req-c8ab6928-28f5-4864-a1c5-09783c0058da tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: d866dcf0-c08b-4b12-be83-8de3e69c439f] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 865.078494] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Expecting reply to msg 55ec1113255d47f982e246dd23b33513 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 865.139967] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] Expecting reply to msg 09de7a97f1a147508bf6ca0d07addf61 in queue reply_4316ed433d59463c9660e50df524570b [ 865.221582] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 55ec1113255d47f982e246dd23b33513 [ 865.278256] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 09de7a97f1a147508bf6ca0d07addf61 [ 865.581116] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 865.592763] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 865.592982] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 865.593146] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 865.615614] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 865.615831] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 865.615997] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 865.616369] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 865.616553] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 865.616710] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 865.623486] nova-conductor[54692]: DEBUG nova.quota [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Getting quotas for project 204d8f668a894827994c416d74f34c18. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 865.625914] nova-conductor[54692]: DEBUG nova.quota [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Getting quotas for user ea7e09048589421e96a4d19f8e884c8a and project 204d8f668a894827994c416d74f34c18. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 865.630539] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: c1001bab-08b9-4bb9-b737-ba7fa5f929f0] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 865.630934] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 865.631130] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 865.631290] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 865.636156] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: c1001bab-08b9-4bb9-b737-ba7fa5f929f0] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 865.636769] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 865.636985] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 865.637150] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 865.649659] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 865.650064] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 865.650256] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 865.650432] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 865.665971] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 865.666391] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 865.666723] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 865.689240] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 865.689676] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 865.690022] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 865.690547] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 865.690912] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 865.691217] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 865.699107] nova-conductor[54691]: DEBUG nova.quota [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] Getting quotas for project 53cc41eb847d485196f118421cb1505a. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 865.701410] nova-conductor[54691]: DEBUG nova.quota [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] Getting quotas for user 14d7167d2fde43338b707199245090e9 and project 53cc41eb847d485196f118421cb1505a. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 865.706304] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] [instance: e7d784c2-53f8-4da7-8c25-4bdf5362e94d] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 865.706912] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 865.707278] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 865.707609] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 865.710595] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] [instance: e7d784c2-53f8-4da7-8c25-4bdf5362e94d] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 865.711438] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 865.711843] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 865.712177] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 865.728618] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 865.729039] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 865.729367] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-f2402f43-648c-4532-8014-e754d1e35a23 tempest-ServersAaction247Test-462353810 tempest-ServersAaction247Test-462353810-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 867.005292] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: ddcf62db-af40-49d2-9465-66a10f0f2e2d] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 205ec1e4-aa48-4292-9a63-d00ef691588a, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance ddcf62db-af40-49d2-9465-66a10f0f2e2d was re-scheduled: Binding failed for port 205ec1e4-aa48-4292-9a63-d00ef691588a, please check neutron logs for more information.\n'] [ 867.005962] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 867.006296] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ddcf62db-af40-49d2-9465-66a10f0f2e2d.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ddcf62db-af40-49d2-9465-66a10f0f2e2d. [ 867.006608] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: ddcf62db-af40-49d2-9465-66a10f0f2e2d] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ddcf62db-af40-49d2-9465-66a10f0f2e2d. [ 867.028208] nova-conductor[54692]: DEBUG nova.network.neutron [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: ddcf62db-af40-49d2-9465-66a10f0f2e2d] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 867.045128] nova-conductor[54692]: DEBUG nova.network.neutron [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: ddcf62db-af40-49d2-9465-66a10f0f2e2d] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 867.048432] nova-conductor[54692]: DEBUG nova.network.neutron [None req-2b284dc3-18b8-4c53-9263-484078d72793 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: ddcf62db-af40-49d2-9465-66a10f0f2e2d] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 868.224844] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Expecting reply to msg 4cf9adb303234bad8dbe52eb8b4cff08 in queue reply_4316ed433d59463c9660e50df524570b [ 868.353507] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 4cf9adb303234bad8dbe52eb8b4cff08 [ 868.727793] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 868.741172] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 868.741479] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 868.741741] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 868.789295] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 868.789435] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 868.789571] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 868.789931] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 868.790114] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 868.790270] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 868.799576] nova-conductor[54691]: DEBUG nova.quota [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Getting quotas for project 721d9e362be84a33aa7956177650d445. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 868.808887] nova-conductor[54691]: DEBUG nova.quota [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Getting quotas for user 5b785ade05de40c88f4bf3ef92e5a25f and project 721d9e362be84a33aa7956177650d445. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 868.814775] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: c42bf89c-91eb-4ed6-90b3-dbdc7b92b7a5] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 868.815500] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 868.815728] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 868.816015] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 868.820081] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: c42bf89c-91eb-4ed6-90b3-dbdc7b92b7a5] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 868.820777] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 868.820978] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 868.821221] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 868.837190] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 868.837547] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 868.837746] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 872.697729] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] [instance: 9f51bbbd-a116-4ebb-b286-4f7db9dbeedb] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 08ad7f7a-3cb5-49b9-a534-2b18dbad7bc8, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 9f51bbbd-a116-4ebb-b286-4f7db9dbeedb was re-scheduled: Binding failed for port 08ad7f7a-3cb5-49b9-a534-2b18dbad7bc8, please check neutron logs for more information.\n'] [ 872.698333] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 872.698564] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9f51bbbd-a116-4ebb-b286-4f7db9dbeedb.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9f51bbbd-a116-4ebb-b286-4f7db9dbeedb. [ 872.698869] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] [instance: 9f51bbbd-a116-4ebb-b286-4f7db9dbeedb] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9f51bbbd-a116-4ebb-b286-4f7db9dbeedb. [ 872.718462] nova-conductor[54691]: DEBUG nova.network.neutron [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] [instance: 9f51bbbd-a116-4ebb-b286-4f7db9dbeedb] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 872.738646] nova-conductor[54691]: DEBUG nova.network.neutron [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] [instance: 9f51bbbd-a116-4ebb-b286-4f7db9dbeedb] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 872.741584] nova-conductor[54691]: DEBUG nova.network.neutron [None req-6d093002-e90d-432e-b9f5-2bf65da037d1 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] [instance: 9f51bbbd-a116-4ebb-b286-4f7db9dbeedb] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 873.332195] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=415,nova_cell1:UPDATE=213,nova_cell1:SAVEPOINT=35,nova_cell1:RELEASE=35,nova_cell1:INSERT=26 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 875.621467] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell0:SELECT=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 875.627826] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Expecting reply to msg 418c266ec64e4f998ac09eec954c5c58 in queue reply_4316ed433d59463c9660e50df524570b [ 875.639948] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=3,nova_cell1:INSERT=7 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 875.647114] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=7,nova_api:UPDATE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 875.648692] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=5,nova_api:DELETE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 875.745882] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 418c266ec64e4f998ac09eec954c5c58 [ 876.130855] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 876.143720] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 876.143941] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 876.144128] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 876.171746] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 876.171987] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 876.172324] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 876.172710] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 876.172897] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 876.173055] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 876.182649] nova-conductor[54691]: DEBUG nova.quota [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Getting quotas for project e049414f9871493398a8e5a07c3ef938. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 876.186170] nova-conductor[54691]: DEBUG nova.quota [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Getting quotas for user 9825cb6e55f8419299e0b4f909b722d6 and project e049414f9871493398a8e5a07c3ef938. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 876.191042] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] [instance: 6add8218-734e-4181-8ada-4657331024e5] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 876.191458] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 876.191968] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 876.192161] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 876.196251] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] [instance: 6add8218-734e-4181-8ada-4657331024e5] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 876.196854] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 876.197145] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 876.197314] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 876.213194] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 876.213393] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 876.213559] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 876.948036] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: 1f0cc9c9-0d9f-40ea-a2cf-ff94c793ff4e] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 6d2239ac-5dcb-4286-baf3-2cb4fd85180d, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 1f0cc9c9-0d9f-40ea-a2cf-ff94c793ff4e was re-scheduled: Binding failed for port 6d2239ac-5dcb-4286-baf3-2cb4fd85180d, please check neutron logs for more information.\n'] [ 876.948372] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 876.948493] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1f0cc9c9-0d9f-40ea-a2cf-ff94c793ff4e.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1f0cc9c9-0d9f-40ea-a2cf-ff94c793ff4e. [ 876.948723] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: 1f0cc9c9-0d9f-40ea-a2cf-ff94c793ff4e] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1f0cc9c9-0d9f-40ea-a2cf-ff94c793ff4e. [ 876.971253] nova-conductor[54692]: DEBUG nova.network.neutron [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: 1f0cc9c9-0d9f-40ea-a2cf-ff94c793ff4e] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 877.008916] nova-conductor[54692]: DEBUG nova.network.neutron [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: 1f0cc9c9-0d9f-40ea-a2cf-ff94c793ff4e] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 877.012769] nova-conductor[54692]: DEBUG nova.network.neutron [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: 1f0cc9c9-0d9f-40ea-a2cf-ff94c793ff4e] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 877.617702] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:UPDATE=193,nova_cell1:SELECT=392,nova_cell1:SAVEPOINT=44,nova_cell1:RELEASE=44,nova_cell1:INSERT=30,nova_cell1:DELETE=2 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 878.523308] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: 1f90a84e-6c32-4846-8908-128323e33e98] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 8892036c-2583-407d-8dd1-b44a5d64495f, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 1f90a84e-6c32-4846-8908-128323e33e98 was re-scheduled: Binding failed for port 8892036c-2583-407d-8dd1-b44a5d64495f, please check neutron logs for more information.\n'] [ 878.523876] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 878.524460] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1f90a84e-6c32-4846-8908-128323e33e98.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1f90a84e-6c32-4846-8908-128323e33e98. [ 878.524635] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: 1f90a84e-6c32-4846-8908-128323e33e98] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1f90a84e-6c32-4846-8908-128323e33e98. [ 878.549794] nova-conductor[54692]: DEBUG nova.network.neutron [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: 1f90a84e-6c32-4846-8908-128323e33e98] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 878.566425] nova-conductor[54692]: DEBUG nova.network.neutron [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: 1f90a84e-6c32-4846-8908-128323e33e98] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 878.571977] nova-conductor[54692]: DEBUG nova.network.neutron [None req-4a1e7b55-b8b9-4ba4-9d19-cfcdb42bb5c8 tempest-MultipleCreateTestJSON-781183485 tempest-MultipleCreateTestJSON-781183485-project-member] [instance: 1f90a84e-6c32-4846-8908-128323e33e98] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 882.735642] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] [instance: e2a1c5a7-34a1-4520-9b37-70f6087f0314] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 1dd6ec9b-97da-4ae2-a996-2fe67cf26328, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance e2a1c5a7-34a1-4520-9b37-70f6087f0314 was re-scheduled: Binding failed for port 1dd6ec9b-97da-4ae2-a996-2fe67cf26328, please check neutron logs for more information.\n'] [ 882.736385] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 882.736633] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance e2a1c5a7-34a1-4520-9b37-70f6087f0314.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance e2a1c5a7-34a1-4520-9b37-70f6087f0314. [ 882.736953] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] [instance: e2a1c5a7-34a1-4520-9b37-70f6087f0314] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance e2a1c5a7-34a1-4520-9b37-70f6087f0314. [ 882.762535] nova-conductor[54691]: DEBUG nova.network.neutron [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] [instance: e2a1c5a7-34a1-4520-9b37-70f6087f0314] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 882.783829] nova-conductor[54691]: DEBUG nova.network.neutron [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] [instance: e2a1c5a7-34a1-4520-9b37-70f6087f0314] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 882.787363] nova-conductor[54691]: DEBUG nova.network.neutron [None req-25b8c6eb-5bb3-4288-8541-76fd3b6fe1b4 tempest-ServersTestMultiNic-922746751 tempest-ServersTestMultiNic-922746751-project-member] [instance: e2a1c5a7-34a1-4520-9b37-70f6087f0314] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 886.177349] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell0:SELECT=5 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 886.199689] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:SELECT=15,nova_cell1:SAVEPOINT=2,nova_cell1:INSERT=37,nova_cell1:RELEASE=2 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 886.210506] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=32,nova_api:UPDATE=4,nova_api:DELETE=2 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 886.212263] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=29,nova_api:UPDATE=1,nova_api:DELETE=3 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 896.938759] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] [instance: 7d7cc825-7dcc-4b21-bf4c-886de451863a] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 1512881e-a7b1-4caf-adc3-f4d2e248d3cc, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 7d7cc825-7dcc-4b21-bf4c-886de451863a was re-scheduled: Binding failed for port 1512881e-a7b1-4caf-adc3-f4d2e248d3cc, please check neutron logs for more information.\n'] [ 896.939318] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 896.939702] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 7d7cc825-7dcc-4b21-bf4c-886de451863a.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 7d7cc825-7dcc-4b21-bf4c-886de451863a. [ 896.939999] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] [instance: 7d7cc825-7dcc-4b21-bf4c-886de451863a] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 7d7cc825-7dcc-4b21-bf4c-886de451863a. [ 896.968065] nova-conductor[54692]: DEBUG nova.network.neutron [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] [instance: 7d7cc825-7dcc-4b21-bf4c-886de451863a] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 897.013252] nova-conductor[54692]: DEBUG nova.network.neutron [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] [instance: 7d7cc825-7dcc-4b21-bf4c-886de451863a] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 897.016760] nova-conductor[54692]: DEBUG nova.network.neutron [None req-34660ada-fb48-4513-af27-bf59e1bb698a tempest-ServerAddressesTestJSON-1247546411 tempest-ServerAddressesTestJSON-1247546411-project-member] [instance: 7d7cc825-7dcc-4b21-bf4c-886de451863a] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 900.943591] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] [instance: 86be55c9-82d5-4a34-b628-4729c42d83c1] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 7f9f039c-f6fc-4bc1-a2fa-317383f60164, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 86be55c9-82d5-4a34-b628-4729c42d83c1 was re-scheduled: Binding failed for port 7f9f039c-f6fc-4bc1-a2fa-317383f60164, please check neutron logs for more information.\n'] [ 900.945178] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 900.945446] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 86be55c9-82d5-4a34-b628-4729c42d83c1.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 86be55c9-82d5-4a34-b628-4729c42d83c1. [ 900.945658] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] [instance: 86be55c9-82d5-4a34-b628-4729c42d83c1] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 86be55c9-82d5-4a34-b628-4729c42d83c1. [ 900.964692] nova-conductor[54691]: DEBUG nova.network.neutron [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] [instance: 86be55c9-82d5-4a34-b628-4729c42d83c1] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 900.979052] nova-conductor[54691]: DEBUG nova.network.neutron [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] [instance: 86be55c9-82d5-4a34-b628-4729c42d83c1] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 900.981728] nova-conductor[54691]: DEBUG nova.network.neutron [None req-8282e491-1f8c-44fe-8756-ffd5bf3244f3 tempest-ServerMetadataNegativeTestJSON-1390807253 tempest-ServerMetadataNegativeTestJSON-1390807253-project-member] [instance: 86be55c9-82d5-4a34-b628-4729c42d83c1] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 906.510524] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Expecting reply to msg 6f3c967d70c8471893d6ae553d980d47 in queue reply_4316ed433d59463c9660e50df524570b [ 906.648795] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 6f3c967d70c8471893d6ae553d980d47 [ 907.012753] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 907.025835] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 907.026088] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 907.026257] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 907.058217] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 907.058435] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 907.058595] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 907.058996] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 907.059183] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 907.059342] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 907.067647] nova-conductor[54691]: DEBUG nova.quota [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Getting quotas for project e1192ea399b74ef29c41ac66a6e23bca. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 907.070653] nova-conductor[54691]: DEBUG nova.quota [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Getting quotas for user f9be2719bf36444a9255ca8c15292c05 and project e1192ea399b74ef29c41ac66a6e23bca. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 907.076229] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] [instance: 41e8e7c0-a518-43bc-82fa-5f77704f4fc1] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 907.076670] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 907.076880] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 907.077046] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 907.084281] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] [instance: 41e8e7c0-a518-43bc-82fa-5f77704f4fc1] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 907.084949] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 907.085140] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 907.085325] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 907.098868] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 907.099108] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 907.099283] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 907.298054] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: 53625c0d-203a-45f0-b899-5d2673413a53] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 7b9d929d-0980-4cd6-ad6b-83398958648e, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 53625c0d-203a-45f0-b899-5d2673413a53 was re-scheduled: Binding failed for port 7b9d929d-0980-4cd6-ad6b-83398958648e, please check neutron logs for more information.\n'] [ 907.298711] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 907.298946] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 53625c0d-203a-45f0-b899-5d2673413a53.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 53625c0d-203a-45f0-b899-5d2673413a53. [ 907.299255] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: 53625c0d-203a-45f0-b899-5d2673413a53] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 53625c0d-203a-45f0-b899-5d2673413a53. [ 907.342792] nova-conductor[54691]: DEBUG nova.network.neutron [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: 53625c0d-203a-45f0-b899-5d2673413a53] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 907.365180] nova-conductor[54691]: DEBUG nova.network.neutron [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: 53625c0d-203a-45f0-b899-5d2673413a53] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 907.369060] nova-conductor[54691]: DEBUG nova.network.neutron [None req-a77a5f6a-199d-446e-80f4-84a60856a1b8 tempest-ImagesTestJSON-1235141815 tempest-ImagesTestJSON-1235141815-project-member] [instance: 53625c0d-203a-45f0-b899-5d2673413a53] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 911.219688] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] [instance: f7ed0a84-a3ca-46f8-8cfb-da2311c9cb90] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port f5acee40-6e6c-4fea-a969-034bb88c660e, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance f7ed0a84-a3ca-46f8-8cfb-da2311c9cb90 was re-scheduled: Binding failed for port f5acee40-6e6c-4fea-a969-034bb88c660e, please check neutron logs for more information.\n'] [ 911.240312] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 911.240312] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f7ed0a84-a3ca-46f8-8cfb-da2311c9cb90.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f7ed0a84-a3ca-46f8-8cfb-da2311c9cb90. [ 911.240312] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] [instance: f7ed0a84-a3ca-46f8-8cfb-da2311c9cb90] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f7ed0a84-a3ca-46f8-8cfb-da2311c9cb90. [ 911.247721] nova-conductor[54691]: DEBUG nova.network.neutron [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] [instance: f7ed0a84-a3ca-46f8-8cfb-da2311c9cb90] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 911.266250] nova-conductor[54691]: DEBUG nova.network.neutron [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] [instance: f7ed0a84-a3ca-46f8-8cfb-da2311c9cb90] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 911.269186] nova-conductor[54691]: DEBUG nova.network.neutron [None req-353a033c-9052-4537-872c-291d68108f1a tempest-VolumesAdminNegativeTest-1053254376 tempest-VolumesAdminNegativeTest-1053254376-project-member] [instance: f7ed0a84-a3ca-46f8-8cfb-da2311c9cb90] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 913.001134] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] [instance: 570ee4e0-caeb-41f9-82d2-fa009f23581d] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 57fb7c5e-9fe9-48df-ac1b-5a913e230491, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 570ee4e0-caeb-41f9-82d2-fa009f23581d was re-scheduled: Binding failed for port 57fb7c5e-9fe9-48df-ac1b-5a913e230491, please check neutron logs for more information.\n'] [ 913.001897] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 913.002244] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 570ee4e0-caeb-41f9-82d2-fa009f23581d.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 570ee4e0-caeb-41f9-82d2-fa009f23581d. [ 913.002588] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] [instance: 570ee4e0-caeb-41f9-82d2-fa009f23581d] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 570ee4e0-caeb-41f9-82d2-fa009f23581d. [ 913.052523] nova-conductor[54692]: DEBUG nova.network.neutron [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] [instance: 570ee4e0-caeb-41f9-82d2-fa009f23581d] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 913.071071] nova-conductor[54692]: DEBUG nova.network.neutron [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] [instance: 570ee4e0-caeb-41f9-82d2-fa009f23581d] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 913.074655] nova-conductor[54692]: DEBUG nova.network.neutron [None req-ce360136-9a77-4eae-9882-377646ff5edd tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] [instance: 570ee4e0-caeb-41f9-82d2-fa009f23581d] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 914.706703] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] [instance: 7d047648-facb-4d58-acf7-a99a3f7adf65] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 53a6658a-47e5-46fc-bdd1-b2efc0b47597, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 7d047648-facb-4d58-acf7-a99a3f7adf65 was re-scheduled: Binding failed for port 53a6658a-47e5-46fc-bdd1-b2efc0b47597, please check neutron logs for more information.\n'] [ 914.707307] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 914.712292] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 7d047648-facb-4d58-acf7-a99a3f7adf65.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 7d047648-facb-4d58-acf7-a99a3f7adf65. [ 914.712583] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] [instance: 7d047648-facb-4d58-acf7-a99a3f7adf65] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 7d047648-facb-4d58-acf7-a99a3f7adf65. [ 914.737483] nova-conductor[54692]: DEBUG nova.network.neutron [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] [instance: 7d047648-facb-4d58-acf7-a99a3f7adf65] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 914.772279] nova-conductor[54692]: DEBUG nova.network.neutron [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] [instance: 7d047648-facb-4d58-acf7-a99a3f7adf65] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 914.788470] nova-conductor[54692]: DEBUG nova.network.neutron [None req-179b36cc-f462-46f0-80f6-f565d4e6c368 tempest-ServerRescueTestJSON-1048251529 tempest-ServerRescueTestJSON-1048251529-project-member] [instance: 7d047648-facb-4d58-acf7-a99a3f7adf65] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 915.146873] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Expecting reply to msg b4143e2ce5134451b69b9463f7dbc9fc in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 915.276524] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg b4143e2ce5134451b69b9463f7dbc9fc [ 915.651538] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 915.664630] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 915.665124] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 915.665124] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 915.698396] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 915.698830] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 915.699150] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 915.699693] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 915.700044] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 915.700366] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 915.713094] nova-conductor[54692]: DEBUG nova.quota [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Getting quotas for project 8003d16b1cc048bc94407d028838a4f6. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 915.717023] nova-conductor[54692]: DEBUG nova.quota [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Getting quotas for user cd65b3ab163145e1bf370aa0dc63f65c and project 8003d16b1cc048bc94407d028838a4f6. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 915.722921] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] [instance: d02ada4d-6801-4353-a664-779c3cd5f163] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 915.723673] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 915.724226] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 915.724506] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 915.731166] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] [instance: d02ada4d-6801-4353-a664-779c3cd5f163] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 915.731166] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 915.731166] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 915.731166] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 915.750484] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 915.750824] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 915.751167] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 917.062787] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell0:SELECT=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 917.087541] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:SELECT=3,nova_cell1:SAVEPOINT=1,nova_cell1:INSERT=8,nova_cell1:RELEASE=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 917.094782] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=6 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 917.097054] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=7,nova_api:UPDATE=1,nova_api:DELETE=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 918.823855] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] [instance: fa27cfe2-6f7f-4397-bcc5-147d6865f404] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 343f61a7-4e93-40f6-87ad-1e52b67a570e, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance fa27cfe2-6f7f-4397-bcc5-147d6865f404 was re-scheduled: Binding failed for port 343f61a7-4e93-40f6-87ad-1e52b67a570e, please check neutron logs for more information.\n'] [ 918.824561] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 918.824733] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance fa27cfe2-6f7f-4397-bcc5-147d6865f404.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance fa27cfe2-6f7f-4397-bcc5-147d6865f404. [ 918.825039] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] [instance: fa27cfe2-6f7f-4397-bcc5-147d6865f404] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance fa27cfe2-6f7f-4397-bcc5-147d6865f404. [ 918.862173] nova-conductor[54692]: DEBUG nova.network.neutron [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] [instance: fa27cfe2-6f7f-4397-bcc5-147d6865f404] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 918.889149] nova-conductor[54692]: DEBUG nova.network.neutron [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] [instance: fa27cfe2-6f7f-4397-bcc5-147d6865f404] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 918.892904] nova-conductor[54692]: DEBUG nova.network.neutron [None req-c6af30f2-763c-48ee-8cbc-70cd6ebca490 tempest-AttachInterfacesTestJSON-221845540 tempest-AttachInterfacesTestJSON-221845540-project-member] [instance: fa27cfe2-6f7f-4397-bcc5-147d6865f404] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 919.827527] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Expecting reply to msg b5730f8a71ac41988eda321e1256cb4c in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 919.964698] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg b5730f8a71ac41988eda321e1256cb4c [ 920.335354] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 920.349846] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 920.350185] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 920.350429] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 920.398257] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 920.398541] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 920.398814] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 920.399257] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 920.399551] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 920.399810] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 920.415212] nova-conductor[54692]: DEBUG nova.quota [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Getting quotas for project 96e09f148ccf440e856165d451068702. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 920.418724] nova-conductor[54692]: DEBUG nova.quota [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Getting quotas for user 246365bf1682441ab86b7b37c2f97971 and project 96e09f148ccf440e856165d451068702. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 920.426191] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] [instance: ca7fc450-351a-407f-96c0-21a39d037e54] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 920.426821] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 920.427089] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 920.427315] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 920.433034] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] [instance: ca7fc450-351a-407f-96c0-21a39d037e54] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 920.433977] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 920.435661] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.002s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 920.436123] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 920.466711] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 920.466949] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.005s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 920.467117] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 922.220358] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] [instance: 768cf3b2-227f-463a-b192-79c50874cded] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 68d97bd5-2b32-4ce1-bb1d-5d90628a1951, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 768cf3b2-227f-463a-b192-79c50874cded was re-scheduled: Binding failed for port 68d97bd5-2b32-4ce1-bb1d-5d90628a1951, please check neutron logs for more information.\n'] [ 922.220986] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 922.221256] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 768cf3b2-227f-463a-b192-79c50874cded.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 768cf3b2-227f-463a-b192-79c50874cded. [ 922.221525] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] [instance: 768cf3b2-227f-463a-b192-79c50874cded] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 768cf3b2-227f-463a-b192-79c50874cded. [ 922.245289] nova-conductor[54692]: DEBUG nova.network.neutron [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] [instance: 768cf3b2-227f-463a-b192-79c50874cded] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 922.260611] nova-conductor[54692]: DEBUG nova.network.neutron [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] [instance: 768cf3b2-227f-463a-b192-79c50874cded] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 922.263086] nova-conductor[54692]: DEBUG nova.network.neutron [None req-adf97f78-5ff9-49ac-8a0e-a7457993d4b3 tempest-ServerGroupTestJSON-28427255 tempest-ServerGroupTestJSON-28427255-project-member] [instance: 768cf3b2-227f-463a-b192-79c50874cded] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 924.008072] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 1f1dd01d-006d-4569-8684-e1eb913ba5d4] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port a06f0a41-bdaf-4090-bfe5-9dc6efaf2200, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 1f1dd01d-006d-4569-8684-e1eb913ba5d4 was re-scheduled: Binding failed for port a06f0a41-bdaf-4090-bfe5-9dc6efaf2200, please check neutron logs for more information.\n'] [ 924.008766] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 924.009000] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1f1dd01d-006d-4569-8684-e1eb913ba5d4.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1f1dd01d-006d-4569-8684-e1eb913ba5d4. [ 924.009213] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 1f1dd01d-006d-4569-8684-e1eb913ba5d4] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1f1dd01d-006d-4569-8684-e1eb913ba5d4. [ 924.034376] nova-conductor[54692]: DEBUG nova.network.neutron [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 1f1dd01d-006d-4569-8684-e1eb913ba5d4] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 924.066942] nova-conductor[54692]: DEBUG nova.network.neutron [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 1f1dd01d-006d-4569-8684-e1eb913ba5d4] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 924.078208] nova-conductor[54692]: DEBUG nova.network.neutron [None req-a7d0de56-06ae-4508-808f-7a924de4bbc9 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 1f1dd01d-006d-4569-8684-e1eb913ba5d4] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 925.123110] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-fd6339b8-5e88-4567-98f6-4fcdf69dc3de tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Expecting reply to msg 5573fd836d524506bcae1c26e08bff1f in queue reply_4316ed433d59463c9660e50df524570b [ 925.272849] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 5573fd836d524506bcae1c26e08bff1f [ 925.625214] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-fd6339b8-5e88-4567-98f6-4fcdf69dc3de tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 925.631907] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-fd6339b8-5e88-4567-98f6-4fcdf69dc3de tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 92700d6b-af86-4870-a94c-56328fc44d9c] While scheduling instance, the build request was already deleted. {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1706}} [ 925.781903] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fd6339b8-5e88-4567-98f6-4fcdf69dc3de tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 925.782106] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fd6339b8-5e88-4567-98f6-4fcdf69dc3de tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 925.782273] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fd6339b8-5e88-4567-98f6-4fcdf69dc3de tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 925.782652] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fd6339b8-5e88-4567-98f6-4fcdf69dc3de tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 925.782843] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fd6339b8-5e88-4567-98f6-4fcdf69dc3de tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 925.782999] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-fd6339b8-5e88-4567-98f6-4fcdf69dc3de tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 925.794265] nova-conductor[54691]: DEBUG nova.quota [None req-fd6339b8-5e88-4567-98f6-4fcdf69dc3de tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Getting quotas for project adf774b7c36f46219e7dfdec663fe624. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 925.797277] nova-conductor[54691]: DEBUG nova.quota [None req-fd6339b8-5e88-4567-98f6-4fcdf69dc3de tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Getting quotas for user 949bec663bc44ef3b1d3eb8c8f6912a9 and project adf774b7c36f46219e7dfdec663fe624. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 925.892437] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Expecting reply to msg ce42579a6d3248ed9bc5f2eaf7129b3c in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 926.017863] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg ce42579a6d3248ed9bc5f2eaf7129b3c [ 926.401171] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 926.422228] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 926.422394] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 926.422520] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 926.475440] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 926.475661] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 926.475824] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 926.476221] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 926.476399] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 926.476589] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 926.505221] nova-conductor[54692]: DEBUG nova.quota [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Getting quotas for project adf774b7c36f46219e7dfdec663fe624. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 926.516605] nova-conductor[54692]: DEBUG nova.quota [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Getting quotas for user 949bec663bc44ef3b1d3eb8c8f6912a9 and project adf774b7c36f46219e7dfdec663fe624. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 926.522350] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 98ce815b-3e40-424f-bad6-a1b6a49e0a74] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 926.522838] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 926.523105] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 926.523293] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 926.544176] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 98ce815b-3e40-424f-bad6-a1b6a49e0a74] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 926.544858] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 926.545070] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 926.545238] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 926.583713] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 926.583937] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 926.584137] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 927.978889] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: c1001bab-08b9-4bb9-b737-ba7fa5f929f0] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port f4444a92-d000-41ce-be26-294346c5047b, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance c1001bab-08b9-4bb9-b737-ba7fa5f929f0 was re-scheduled: Binding failed for port f4444a92-d000-41ce-be26-294346c5047b, please check neutron logs for more information.\n'] [ 927.979675] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 927.979868] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c1001bab-08b9-4bb9-b737-ba7fa5f929f0.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c1001bab-08b9-4bb9-b737-ba7fa5f929f0. [ 927.980177] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: c1001bab-08b9-4bb9-b737-ba7fa5f929f0] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c1001bab-08b9-4bb9-b737-ba7fa5f929f0. [ 928.008143] nova-conductor[54692]: DEBUG nova.network.neutron [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: c1001bab-08b9-4bb9-b737-ba7fa5f929f0] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 928.028318] nova-conductor[54692]: DEBUG nova.network.neutron [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: c1001bab-08b9-4bb9-b737-ba7fa5f929f0] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 928.031879] nova-conductor[54692]: DEBUG nova.network.neutron [None req-48b8b4a5-2b80-41af-8c09-a99e1346c60b tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: c1001bab-08b9-4bb9-b737-ba7fa5f929f0] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 929.300662] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Expecting reply to msg ff0bd0074ddf4c1ba763edf0837d5768 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 929.419921] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg ff0bd0074ddf4c1ba763edf0837d5768 [ 929.808427] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 929.816096] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 929.816415] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 929.816644] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 929.847769] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 929.848071] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 929.848382] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 929.848835] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 929.849047] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 929.849661] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 929.861903] nova-conductor[54692]: DEBUG nova.quota [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Getting quotas for project 204d8f668a894827994c416d74f34c18. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 929.865142] nova-conductor[54692]: DEBUG nova.quota [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Getting quotas for user ea7e09048589421e96a4d19f8e884c8a and project 204d8f668a894827994c416d74f34c18. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 929.870607] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: 65fc24cb-71af-4ca3-8e46-b010a79a65be] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 929.871156] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 929.871424] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 929.871657] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 929.874993] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: 65fc24cb-71af-4ca3-8e46-b010a79a65be] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 929.875691] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 929.875969] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 929.876220] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 929.891671] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 929.891961] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 929.892341] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 932.534099] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Expecting reply to msg 237caf0c936a4f1f8f21fc9fcc55be89 in queue reply_4316ed433d59463c9660e50df524570b [ 932.655698] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 237caf0c936a4f1f8f21fc9fcc55be89 [ 933.038235] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 933.058100] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 933.058389] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 933.058608] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 933.111356] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 933.111718] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 933.111971] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 933.112424] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 933.112658] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 933.112866] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 933.121795] nova-conductor[54691]: DEBUG nova.quota [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Getting quotas for project 37d11ba4c5e44dc69920c21d01aee53d. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 933.124927] nova-conductor[54691]: DEBUG nova.quota [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Getting quotas for user 244ec553c8df4c1d9789386e00626390 and project 37d11ba4c5e44dc69920c21d01aee53d. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 933.130260] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] [instance: 75d0ffcb-0478-441d-a2dd-0429d5d03055] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 933.130848] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 933.131102] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 933.131313] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 933.136665] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] [instance: 75d0ffcb-0478-441d-a2dd-0429d5d03055] block_device_mapping [BlockDeviceMapping(attachment_id=05b10067-23e1-4bd5-accf-23207b401b15,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='volume',device_name=None,device_type=None,disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id=None,instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='volume',tag=None,updated_at=,uuid=,volume_id='e058b35d-a55c-4cbe-8ab6-8dd59305b59d',volume_size=1,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 933.137377] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 933.137642] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 933.137840] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 933.161148] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 933.161337] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 933.161547] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 933.696583] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=374,nova_cell1:UPDATE=195,nova_cell1:INSERT=33,nova_cell1:SAVEPOINT=41,nova_cell1:RELEASE=41 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 936.354766] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: c42bf89c-91eb-4ed6-90b3-dbdc7b92b7a5] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 3a7327d4-0399-4b0e-ba0d-a4cab0550f4f, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance c42bf89c-91eb-4ed6-90b3-dbdc7b92b7a5 was re-scheduled: Binding failed for port 3a7327d4-0399-4b0e-ba0d-a4cab0550f4f, please check neutron logs for more information.\n'] [ 936.355324] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 936.355550] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c42bf89c-91eb-4ed6-90b3-dbdc7b92b7a5.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c42bf89c-91eb-4ed6-90b3-dbdc7b92b7a5. [ 936.355862] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: c42bf89c-91eb-4ed6-90b3-dbdc7b92b7a5] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c42bf89c-91eb-4ed6-90b3-dbdc7b92b7a5. [ 936.382302] nova-conductor[54692]: DEBUG nova.network.neutron [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: c42bf89c-91eb-4ed6-90b3-dbdc7b92b7a5] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 936.398188] nova-conductor[54692]: DEBUG nova.network.neutron [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: c42bf89c-91eb-4ed6-90b3-dbdc7b92b7a5] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 936.401174] nova-conductor[54692]: DEBUG nova.network.neutron [None req-9457cabf-f3b3-4003-aea5-6e5df03f1c19 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: c42bf89c-91eb-4ed6-90b3-dbdc7b92b7a5] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 937.945415] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Expecting reply to msg cd3824d389744ac4b4a1f2d082a2b5b7 in queue reply_4316ed433d59463c9660e50df524570b [ 938.074432] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg cd3824d389744ac4b4a1f2d082a2b5b7 [ 938.163776] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:SELECT=401,nova_cell1:UPDATE=186,nova_cell1:INSERT=22,nova_cell1:SAVEPOINT=38,nova_cell1:RELEASE=38 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 938.449700] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 938.462479] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 938.462755] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 938.462957] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 938.499083] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 938.499387] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 938.499633] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 938.500638] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 938.500867] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 938.501554] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 938.517841] nova-conductor[54691]: DEBUG nova.quota [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Getting quotas for project 721d9e362be84a33aa7956177650d445. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 938.525891] nova-conductor[54691]: DEBUG nova.quota [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Getting quotas for user 5b785ade05de40c88f4bf3ef92e5a25f and project 721d9e362be84a33aa7956177650d445. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 938.540704] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: d929eb07-7c21-40fe-9de5-da578e068a13] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 938.540704] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 938.540704] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 938.540704] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 938.544609] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: d929eb07-7c21-40fe-9de5-da578e068a13] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 938.544811] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 938.545070] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 938.545310] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 938.567081] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 938.567347] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 938.567576] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 939.854974] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell0:SELECT=4 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 939.877915] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=12,nova_cell1:INSERT=30,nova_cell1:SAVEPOINT=1,nova_cell1:RELEASE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 939.887001] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=23,nova_api:UPDATE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 939.889600] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=25,nova_api:DELETE=4,nova_api:UPDATE=3 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 941.843947] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] [instance: 6add8218-734e-4181-8ada-4657331024e5] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port c6993fc6-8b41-4675-8af3-4c43f6501e88, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 6add8218-734e-4181-8ada-4657331024e5 was re-scheduled: Binding failed for port c6993fc6-8b41-4675-8af3-4c43f6501e88, please check neutron logs for more information.\n'] [ 941.844291] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 941.844596] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 6add8218-734e-4181-8ada-4657331024e5.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 6add8218-734e-4181-8ada-4657331024e5. [ 941.844957] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] [instance: 6add8218-734e-4181-8ada-4657331024e5] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 6add8218-734e-4181-8ada-4657331024e5. [ 941.880319] nova-conductor[54691]: DEBUG nova.network.neutron [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] [instance: 6add8218-734e-4181-8ada-4657331024e5] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 941.896684] nova-conductor[54691]: DEBUG nova.network.neutron [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] [instance: 6add8218-734e-4181-8ada-4657331024e5] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 941.901315] nova-conductor[54691]: DEBUG nova.network.neutron [None req-b362cc11-cd22-4da0-9533-8f59011c8987 tempest-AttachVolumeTestJSON-1836922524 tempest-AttachVolumeTestJSON-1836922524-project-member] [instance: 6add8218-734e-4181-8ada-4657331024e5] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 945.765619] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] [instance: 41e8e7c0-a518-43bc-82fa-5f77704f4fc1] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 0a528b85-3544-4be3-abfa-b5cfea3ebc4f, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 41e8e7c0-a518-43bc-82fa-5f77704f4fc1 was re-scheduled: Binding failed for port 0a528b85-3544-4be3-abfa-b5cfea3ebc4f, please check neutron logs for more information.\n'] [ 945.766166] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 945.766387] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 41e8e7c0-a518-43bc-82fa-5f77704f4fc1.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 41e8e7c0-a518-43bc-82fa-5f77704f4fc1. [ 945.766586] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] [instance: 41e8e7c0-a518-43bc-82fa-5f77704f4fc1] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 41e8e7c0-a518-43bc-82fa-5f77704f4fc1. [ 945.789509] nova-conductor[54691]: DEBUG nova.network.neutron [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] [instance: 41e8e7c0-a518-43bc-82fa-5f77704f4fc1] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 945.805303] nova-conductor[54691]: DEBUG nova.network.neutron [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] [instance: 41e8e7c0-a518-43bc-82fa-5f77704f4fc1] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 945.808440] nova-conductor[54691]: DEBUG nova.network.neutron [None req-7c1701df-8a8b-4d7b-b280-f68ff6f52c16 tempest-ServersNegativeTestJSON-1043856013 tempest-ServersNegativeTestJSON-1043856013-project-member] [instance: 41e8e7c0-a518-43bc-82fa-5f77704f4fc1] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 947.417197] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] [instance: d02ada4d-6801-4353-a664-779c3cd5f163] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port ad725b37-2130-4767-be14-161ff994714f, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance d02ada4d-6801-4353-a664-779c3cd5f163 was re-scheduled: Binding failed for port ad725b37-2130-4767-be14-161ff994714f, please check neutron logs for more information.\n'] [ 947.417754] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 947.417980] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d02ada4d-6801-4353-a664-779c3cd5f163.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d02ada4d-6801-4353-a664-779c3cd5f163. [ 947.418183] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] [instance: d02ada4d-6801-4353-a664-779c3cd5f163] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d02ada4d-6801-4353-a664-779c3cd5f163. [ 947.440686] nova-conductor[54691]: DEBUG nova.network.neutron [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] [instance: d02ada4d-6801-4353-a664-779c3cd5f163] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 947.461071] nova-conductor[54691]: DEBUG nova.network.neutron [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] [instance: d02ada4d-6801-4353-a664-779c3cd5f163] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 947.464342] nova-conductor[54691]: DEBUG nova.network.neutron [None req-5e8d561e-1b49-4914-81c6-5c9f1369c579 tempest-AttachVolumeNegativeTest-1702163546 tempest-AttachVolumeNegativeTest-1702163546-project-member] [instance: d02ada4d-6801-4353-a664-779c3cd5f163] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 948.509944] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell0:SELECT=3 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 948.547714] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:SELECT=7,nova_cell1:SAVEPOINT=1,nova_cell1:INSERT=16,nova_cell1:RELEASE=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 948.562399] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=17,nova_api:DELETE=1,nova_api:UPDATE=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 948.564490] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=16,nova_api:DELETE=2,nova_api:UPDATE=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 951.532813] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] [instance: ca7fc450-351a-407f-96c0-21a39d037e54] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 4a9dd839-ec2d-49fa-a097-d4407ef9bc7e, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance ca7fc450-351a-407f-96c0-21a39d037e54 was re-scheduled: Binding failed for port 4a9dd839-ec2d-49fa-a097-d4407ef9bc7e, please check neutron logs for more information.\n'] [ 951.533950] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 951.534337] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ca7fc450-351a-407f-96c0-21a39d037e54.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ca7fc450-351a-407f-96c0-21a39d037e54. [ 951.534868] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] [instance: ca7fc450-351a-407f-96c0-21a39d037e54] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ca7fc450-351a-407f-96c0-21a39d037e54. [ 951.559713] nova-conductor[54692]: DEBUG nova.network.neutron [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] [instance: ca7fc450-351a-407f-96c0-21a39d037e54] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 951.575849] nova-conductor[54692]: DEBUG nova.network.neutron [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] [instance: ca7fc450-351a-407f-96c0-21a39d037e54] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 951.582347] nova-conductor[54692]: DEBUG nova.network.neutron [None req-1a208bc6-08e3-41ac-a8a8-36b9af58a395 tempest-ServersTestJSON-1369522877 tempest-ServersTestJSON-1369522877-project-member] [instance: ca7fc450-351a-407f-96c0-21a39d037e54] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 955.188785] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 98ce815b-3e40-424f-bad6-a1b6a49e0a74] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 0534a202-0f8f-45f6-b574-5d9aeeed9133, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 98ce815b-3e40-424f-bad6-a1b6a49e0a74 was re-scheduled: Binding failed for port 0534a202-0f8f-45f6-b574-5d9aeeed9133, please check neutron logs for more information.\n'] [ 955.189419] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 955.189735] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 98ce815b-3e40-424f-bad6-a1b6a49e0a74.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 98ce815b-3e40-424f-bad6-a1b6a49e0a74. [ 955.190099] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 98ce815b-3e40-424f-bad6-a1b6a49e0a74] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 98ce815b-3e40-424f-bad6-a1b6a49e0a74. [ 955.213721] nova-conductor[54692]: DEBUG nova.network.neutron [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 98ce815b-3e40-424f-bad6-a1b6a49e0a74] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 955.229231] nova-conductor[54692]: DEBUG nova.network.neutron [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 98ce815b-3e40-424f-bad6-a1b6a49e0a74] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 955.232291] nova-conductor[54692]: DEBUG nova.network.neutron [None req-a45c6be5-e73b-4367-8afb-678632b2df82 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 98ce815b-3e40-424f-bad6-a1b6a49e0a74] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 956.303740] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Expecting reply to msg 1799e7ecf7b7405885422e94bfb867c1 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 956.417361] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 1799e7ecf7b7405885422e94bfb867c1 [ 956.806303] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 956.818817] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 956.819253] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 956.819570] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 956.844895] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: 65fc24cb-71af-4ca3-8e46-b010a79a65be] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port ba08710f-f563-4181-b8d2-4bb9884cb79e, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 65fc24cb-71af-4ca3-8e46-b010a79a65be was re-scheduled: Binding failed for port ba08710f-f563-4181-b8d2-4bb9884cb79e, please check neutron logs for more information.\n'] [ 956.845438] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 956.846042] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 65fc24cb-71af-4ca3-8e46-b010a79a65be.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 65fc24cb-71af-4ca3-8e46-b010a79a65be. [ 956.846042] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: 65fc24cb-71af-4ca3-8e46-b010a79a65be] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 65fc24cb-71af-4ca3-8e46-b010a79a65be. [ 956.858880] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 956.859093] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 956.859271] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 956.859675] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 956.859897] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 956.860099] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 956.877678] nova-conductor[54692]: DEBUG nova.network.neutron [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: 65fc24cb-71af-4ca3-8e46-b010a79a65be] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 956.881006] nova-conductor[54692]: DEBUG nova.quota [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Getting quotas for project adf774b7c36f46219e7dfdec663fe624. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 956.882805] nova-conductor[54692]: DEBUG nova.quota [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Getting quotas for user 949bec663bc44ef3b1d3eb8c8f6912a9 and project adf774b7c36f46219e7dfdec663fe624. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 956.888402] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: fc58f8bf-849a-452a-a1cb-daec65f662a7] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 956.888793] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 956.888988] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 956.889150] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 956.891452] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: fc58f8bf-849a-452a-a1cb-daec65f662a7] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 956.892172] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 956.892270] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 956.892432] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 956.894711] nova-conductor[54692]: DEBUG nova.network.neutron [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: 65fc24cb-71af-4ca3-8e46-b010a79a65be] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 956.899742] nova-conductor[54692]: DEBUG nova.network.neutron [None req-95c7a0ab-fdab-4784-9d6c-9c8d3cff6e54 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: 65fc24cb-71af-4ca3-8e46-b010a79a65be] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 956.913928] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 956.914123] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 956.914286] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 957.943156] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Expecting reply to msg 1bc7b226dd66490aaad08556091fe583 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 958.054803] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 1bc7b226dd66490aaad08556091fe583 [ 958.445986] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 958.460655] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 958.460940] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 958.461143] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 958.492403] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] [instance: 75d0ffcb-0478-441d-a2dd-0429d5d03055] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 8118c02b-1653-43b8-9584-07011b2a208d, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 75d0ffcb-0478-441d-a2dd-0429d5d03055 was re-scheduled: Binding failed for port 8118c02b-1653-43b8-9584-07011b2a208d, please check neutron logs for more information.\n'] [ 958.492403] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 958.492403] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 75d0ffcb-0478-441d-a2dd-0429d5d03055.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 75d0ffcb-0478-441d-a2dd-0429d5d03055. [ 958.492403] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] [instance: 75d0ffcb-0478-441d-a2dd-0429d5d03055] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 75d0ffcb-0478-441d-a2dd-0429d5d03055. [ 958.503196] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 958.503667] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 958.504194] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 958.504716] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 958.505074] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 958.505384] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 958.520028] nova-conductor[54692]: DEBUG nova.quota [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Getting quotas for project 204d8f668a894827994c416d74f34c18. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 958.522647] nova-conductor[54692]: DEBUG nova.quota [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Getting quotas for user ea7e09048589421e96a4d19f8e884c8a and project 204d8f668a894827994c416d74f34c18. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 958.529037] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: effae28c-cbe1-4656-9a92-9829d929e3d5] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 958.529517] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 958.529717] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 958.529879] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 958.541318] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: effae28c-cbe1-4656-9a92-9829d929e3d5] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 958.541986] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 958.542176] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 958.542346] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 958.546751] nova-conductor[54691]: DEBUG nova.network.neutron [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] [instance: 75d0ffcb-0478-441d-a2dd-0429d5d03055] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 958.561312] nova-conductor[54691]: DEBUG nova.network.neutron [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] [instance: 75d0ffcb-0478-441d-a2dd-0429d5d03055] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 958.562456] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 958.562990] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 958.563191] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 958.565928] nova-conductor[54691]: DEBUG nova.network.neutron [None req-e010e025-8ad5-49a2-be8a-b3592d29f5ca tempest-ServersTestBootFromVolume-1046228888 tempest-ServersTestBootFromVolume-1046228888-project-member] [instance: 75d0ffcb-0478-441d-a2dd-0429d5d03055] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 963.325678] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: d929eb07-7c21-40fe-9de5-da578e068a13] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 7e97d758-e942-48f8-9b95-ba51fac25f7c, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance d929eb07-7c21-40fe-9de5-da578e068a13 was re-scheduled: Binding failed for port 7e97d758-e942-48f8-9b95-ba51fac25f7c, please check neutron logs for more information.\n'] [ 963.326278] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 963.326507] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d929eb07-7c21-40fe-9de5-da578e068a13.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d929eb07-7c21-40fe-9de5-da578e068a13. [ 963.326739] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: d929eb07-7c21-40fe-9de5-da578e068a13] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d929eb07-7c21-40fe-9de5-da578e068a13. [ 963.351341] nova-conductor[54691]: DEBUG nova.network.neutron [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: d929eb07-7c21-40fe-9de5-da578e068a13] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 963.373819] nova-conductor[54691]: DEBUG nova.network.neutron [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: d929eb07-7c21-40fe-9de5-da578e068a13] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 963.377400] nova-conductor[54691]: DEBUG nova.network.neutron [None req-e29d3081-b077-4a2a-8665-6ddffc6f1321 tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: d929eb07-7c21-40fe-9de5-da578e068a13] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 964.597837] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Expecting reply to msg 800fd37b020c47b7846550a4477d203c in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 964.727671] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 800fd37b020c47b7846550a4477d203c [ 964.904984] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Expecting reply to msg e05f1ea5c55b4921a28d8995b13eca07 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 965.048173] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg e05f1ea5c55b4921a28d8995b13eca07 [ 965.100264] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 965.115935] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 965.116188] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 965.116356] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 965.154833] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 965.155070] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 965.155236] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 965.155591] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 965.155919] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 965.156079] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 965.172311] nova-conductor[54692]: DEBUG nova.quota [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Getting quotas for project 721d9e362be84a33aa7956177650d445. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 965.177966] nova-conductor[54692]: DEBUG nova.quota [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Getting quotas for user 5b785ade05de40c88f4bf3ef92e5a25f and project 721d9e362be84a33aa7956177650d445. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 965.184768] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: b3a23138-ab20-4c59-9d67-761afd7d4a06] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 965.185255] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 965.185459] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 965.187403] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 965.192542] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: b3a23138-ab20-4c59-9d67-761afd7d4a06] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 965.193349] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 965.193459] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 965.193674] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 965.212137] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 965.212313] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 965.212484] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 965.407944] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 965.418836] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 965.419070] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 965.419239] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 965.473080] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 965.473321] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 965.473467] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 965.473902] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 965.474050] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 965.474212] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 965.490828] nova-conductor[54692]: DEBUG nova.quota [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Getting quotas for project 32c9de32b740446c968d226b5926ecd8. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 965.493788] nova-conductor[54692]: DEBUG nova.quota [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Getting quotas for user ccb09ab46a4a45c3b361cd8a4129d471 and project 32c9de32b740446c968d226b5926ecd8. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 965.501636] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] [instance: 4320d2de-3271-4284-a551-1761e6fcd71e] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 965.502030] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 965.502196] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 965.502363] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 965.506320] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] [instance: 4320d2de-3271-4284-a551-1761e6fcd71e] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 965.507045] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 965.507773] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 965.507773] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 965.523028] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 965.523338] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 965.523691] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 975.479895] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell0:SELECT=4 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 975.509599] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=12,nova_cell1:INSERT=29,nova_cell1:SAVEPOINT=1,nova_cell1:RELEASE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 975.516297] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=25,nova_api:UPDATE=2,nova_api:DELETE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 975.520453] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=23,nova_api:UPDATE=2,nova_api:DELETE=3 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 975.881922] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: fc58f8bf-849a-452a-a1cb-daec65f662a7] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 246ff4b3-1dd3-440f-aafb-c044dc9cb709, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance fc58f8bf-849a-452a-a1cb-daec65f662a7 was re-scheduled: Binding failed for port 246ff4b3-1dd3-440f-aafb-c044dc9cb709, please check neutron logs for more information.\n'] [ 975.882543] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 975.882782] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance fc58f8bf-849a-452a-a1cb-daec65f662a7.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance fc58f8bf-849a-452a-a1cb-daec65f662a7. [ 975.883028] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: fc58f8bf-849a-452a-a1cb-daec65f662a7] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance fc58f8bf-849a-452a-a1cb-daec65f662a7. [ 975.903935] nova-conductor[54691]: DEBUG nova.network.neutron [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: fc58f8bf-849a-452a-a1cb-daec65f662a7] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 975.919227] nova-conductor[54691]: DEBUG nova.network.neutron [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: fc58f8bf-849a-452a-a1cb-daec65f662a7] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 975.922085] nova-conductor[54691]: DEBUG nova.network.neutron [None req-f7ed7e0c-eadb-49c2-92d6-9dc9012a071d tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: fc58f8bf-849a-452a-a1cb-daec65f662a7] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 977.356709] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Expecting reply to msg 04031f6f4906446e8af8dc0843b3877f in queue reply_4316ed433d59463c9660e50df524570b [ 977.393400] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: effae28c-cbe1-4656-9a92-9829d929e3d5] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 26fd6fc5-060f-497f-9d38-23f8d5ec7bb1, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance effae28c-cbe1-4656-9a92-9829d929e3d5 was re-scheduled: Binding failed for port 26fd6fc5-060f-497f-9d38-23f8d5ec7bb1, please check neutron logs for more information.\n'] [ 977.394042] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 977.394315] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance effae28c-cbe1-4656-9a92-9829d929e3d5.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance effae28c-cbe1-4656-9a92-9829d929e3d5. [ 977.394697] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: effae28c-cbe1-4656-9a92-9829d929e3d5] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance effae28c-cbe1-4656-9a92-9829d929e3d5. [ 977.420924] nova-conductor[54692]: DEBUG nova.network.neutron [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: effae28c-cbe1-4656-9a92-9829d929e3d5] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 977.435506] nova-conductor[54692]: DEBUG nova.network.neutron [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: effae28c-cbe1-4656-9a92-9829d929e3d5] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 977.438722] nova-conductor[54692]: DEBUG nova.network.neutron [None req-f72e5c39-6303-4d5c-9d38-770e18357219 tempest-ServersTestJSON-910337478 tempest-ServersTestJSON-910337478-project-member] [instance: effae28c-cbe1-4656-9a92-9829d929e3d5] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 977.486860] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 04031f6f4906446e8af8dc0843b3877f [ 977.860703] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 977.877894] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 977.878455] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 977.878825] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 977.904962] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 977.905448] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 977.905771] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 977.906383] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 977.906707] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 977.907002] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 977.922730] nova-conductor[54691]: DEBUG nova.quota [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Getting quotas for project adf774b7c36f46219e7dfdec663fe624. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 977.926039] nova-conductor[54691]: DEBUG nova.quota [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Getting quotas for user 949bec663bc44ef3b1d3eb8c8f6912a9 and project adf774b7c36f46219e7dfdec663fe624. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 977.931002] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 9f629513-ffbd-4879-949e-816c160149b2] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 977.931610] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 977.931955] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 977.932276] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 977.936839] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 9f629513-ffbd-4879-949e-816c160149b2] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 977.937695] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 977.938038] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 977.938450] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 977.958907] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 977.959326] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 977.959675] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 979.018217] nova-conductor[54691]: ERROR nova.scheduler.utils [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: b3a23138-ab20-4c59-9d67-761afd7d4a06] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port aba92c72-ee92-4337-b5d7-c7ad37a2aae4, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance b3a23138-ab20-4c59-9d67-761afd7d4a06 was re-scheduled: Binding failed for port aba92c72-ee92-4337-b5d7-c7ad37a2aae4, please check neutron logs for more information.\n'] [ 979.018567] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Rescheduling: True {{(pid=54691) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 979.018567] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance b3a23138-ab20-4c59-9d67-761afd7d4a06.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance b3a23138-ab20-4c59-9d67-761afd7d4a06. [ 979.018910] nova-conductor[54691]: WARNING nova.scheduler.utils [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: b3a23138-ab20-4c59-9d67-761afd7d4a06] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance b3a23138-ab20-4c59-9d67-761afd7d4a06. [ 979.046913] nova-conductor[54691]: DEBUG nova.network.neutron [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: b3a23138-ab20-4c59-9d67-761afd7d4a06] deallocate_for_instance() {{(pid=54691) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 979.065577] nova-conductor[54691]: DEBUG nova.network.neutron [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: b3a23138-ab20-4c59-9d67-761afd7d4a06] Instance cache missing network info. {{(pid=54691) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 979.068773] nova-conductor[54691]: DEBUG nova.network.neutron [None req-c0ddec34-58e3-4854-b62b-4b7db05135cd tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: b3a23138-ab20-4c59-9d67-761afd7d4a06] Updating instance_info_cache with network_info: [] {{(pid=54691) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 980.145221] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Expecting reply to msg d1c5309f4d4a4b24ac9744600974682d in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 980.171259] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Expecting reply to msg 516b5871ca304c67bf629a0269096d79 in queue reply_4316ed433d59463c9660e50df524570b [ 980.264522] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg d1c5309f4d4a4b24ac9744600974682d [ 980.277672] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 516b5871ca304c67bf629a0269096d79 [ 980.648050] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 980.659949] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 980.660194] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 980.660361] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 980.674479] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 980.683790] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 980.683994] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 980.684179] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 980.684536] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 980.684708] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 980.684862] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 980.685654] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 980.686023] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 980.686323] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 980.693155] nova-conductor[54692]: DEBUG nova.quota [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Getting quotas for project e2d2c93d1f1542978563392cc99f270d. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 980.695115] nova-conductor[54692]: DEBUG nova.quota [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Getting quotas for user 76f9c81addfc4ddd9962416c42d0a35c and project e2d2c93d1f1542978563392cc99f270d. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 980.700026] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] [instance: 267e30d2-c00c-4edf-8665-cfa88eba0938] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 980.700414] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 980.700606] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 980.700760] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 980.704814] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] [instance: 267e30d2-c00c-4edf-8665-cfa88eba0938] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 980.705385] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 980.705564] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 980.705719] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 980.710714] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 980.711090] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 980.711399] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 980.711923] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 980.712261] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 980.712549] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 980.718894] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 980.718970] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 980.719120] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-cfd7e1b0-284d-47db-8d71-55a9f1e46bb7 tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 980.719957] nova-conductor[54691]: DEBUG nova.quota [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Getting quotas for project 721d9e362be84a33aa7956177650d445. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 980.722029] nova-conductor[54691]: DEBUG nova.quota [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Getting quotas for user 5b785ade05de40c88f4bf3ef92e5a25f and project 721d9e362be84a33aa7956177650d445. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 980.726735] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: 8e17c8c1-e662-4695-9380-a43b7dd4ced6] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 980.727292] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 980.727604] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 980.727878] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 980.730514] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: 8e17c8c1-e662-4695-9380-a43b7dd4ced6] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 980.731258] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 980.731557] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 980.731851] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 980.748693] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 980.748907] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 980.749063] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 980.887000] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] [instance: 4320d2de-3271-4284-a551-1761e6fcd71e] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 4181d88c-19ef-422c-95b4-591cd4136c55, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 4320d2de-3271-4284-a551-1761e6fcd71e was re-scheduled: Binding failed for port 4181d88c-19ef-422c-95b4-591cd4136c55, please check neutron logs for more information.\n'] [ 980.887575] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 980.887837] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 4320d2de-3271-4284-a551-1761e6fcd71e.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 4320d2de-3271-4284-a551-1761e6fcd71e. [ 980.888278] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] [instance: 4320d2de-3271-4284-a551-1761e6fcd71e] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 4320d2de-3271-4284-a551-1761e6fcd71e. [ 980.909580] nova-conductor[54692]: DEBUG nova.network.neutron [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] [instance: 4320d2de-3271-4284-a551-1761e6fcd71e] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 980.924205] nova-conductor[54692]: DEBUG nova.network.neutron [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] [instance: 4320d2de-3271-4284-a551-1761e6fcd71e] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 980.927022] nova-conductor[54692]: DEBUG nova.network.neutron [None req-628cc1df-be75-407d-9f77-1fef73d44376 tempest-ServerActionsTestJSON-972253171 tempest-ServerActionsTestJSON-972253171-project-member] [instance: 4320d2de-3271-4284-a551-1761e6fcd71e] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 990.689637] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell0:SELECT=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 990.706945] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=3,nova_cell1:SAVEPOINT=1,nova_cell1:INSERT=8,nova_cell1:RELEASE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 990.716244] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell0:SELECT=2 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 990.716893] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=5,nova_api:UPDATE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 990.718294] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=7,nova_api:DELETE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 990.735003] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:SELECT=6,nova_cell1:INSERT=14 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 990.742650] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=12,nova_api:UPDATE=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 990.747407] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=12,nova_api:DELETE=2,nova_api:UPDATE=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 992.723190] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 9f629513-ffbd-4879-949e-816c160149b2] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port ced04c10-bd0f-4846-9e80-67425c845633, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 9f629513-ffbd-4879-949e-816c160149b2 was re-scheduled: Binding failed for port ced04c10-bd0f-4846-9e80-67425c845633, please check neutron logs for more information.\n'] [ 992.724329] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 992.725162] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9f629513-ffbd-4879-949e-816c160149b2.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9f629513-ffbd-4879-949e-816c160149b2. [ 992.725625] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 9f629513-ffbd-4879-949e-816c160149b2] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9f629513-ffbd-4879-949e-816c160149b2. [ 992.765831] nova-conductor[54692]: DEBUG nova.network.neutron [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 9f629513-ffbd-4879-949e-816c160149b2] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 992.781473] nova-conductor[54692]: DEBUG nova.network.neutron [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 9f629513-ffbd-4879-949e-816c160149b2] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 992.784364] nova-conductor[54692]: DEBUG nova.network.neutron [None req-3256ee14-e0b8-452f-8842-69b3634f23b4 tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 9f629513-ffbd-4879-949e-816c160149b2] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 993.124577] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-2fa980b1-1f3e-4df1-8376-2a7a4c5f21da tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Targeting cell 2d4e634d-6292-45c4-9eb1-09b106cacc48(cell1) for conductor method rebuild_instance {{(pid=54692) wrapper /opt/stack/nova/nova/conductor/manager.py:95}} [ 993.125916] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2fa980b1-1f3e-4df1-8376-2a7a4c5f21da tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 993.125916] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2fa980b1-1f3e-4df1-8376-2a7a4c5f21da tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 993.125916] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-2fa980b1-1f3e-4df1-8376-2a7a4c5f21da tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 993.132283] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-2fa980b1-1f3e-4df1-8376-2a7a4c5f21da tempest-ServerShowV254Test-392752660 tempest-ServerShowV254Test-392752660-project-member] [instance: 267e30d2-c00c-4edf-8665-cfa88eba0938] No migration record for the rebuild/evacuate request. {{(pid=54692) rebuild_instance /opt/stack/nova/nova/conductor/manager.py:1227}} [ 993.712410] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=360,nova_cell1:UPDATE=187,nova_cell1:SAVEPOINT=33,nova_cell1:RELEASE=33,nova_cell1:DELETE=2,nova_cell1:INSERT=22 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 993.880715] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Expecting reply to msg a1c15e731a1d406faaf3e5539ab87f45 in queue reply_4316ed433d59463c9660e50df524570b [ 993.998842] nova-conductor[54691]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg a1c15e731a1d406faaf3e5539ab87f45 [ 994.383000] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=54691) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 994.396453] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 994.396673] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 994.396838] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 994.426427] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 994.426786] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 994.426786] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 994.427151] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 994.427330] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 994.427489] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 994.436094] nova-conductor[54691]: DEBUG nova.quota [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Getting quotas for project adf774b7c36f46219e7dfdec663fe624. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 994.438811] nova-conductor[54691]: DEBUG nova.quota [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Getting quotas for user 949bec663bc44ef3b1d3eb8c8f6912a9 and project adf774b7c36f46219e7dfdec663fe624. Resources: {'cores', 'ram', 'instances'} {{(pid=54691) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 994.443539] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 5732777e-b491-4e60-bc28-20f3f636f6fe] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54691) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 994.443979] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 994.444193] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 994.444357] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 994.448211] nova-conductor[54691]: DEBUG nova.conductor.manager [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 5732777e-b491-4e60-bc28-20f3f636f6fe] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54691) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 994.448888] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 994.449081] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 994.449241] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 994.463016] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 994.463269] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 994.463470] nova-conductor[54691]: DEBUG oslo_concurrency.lockutils [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54691) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 997.163349] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: 8e17c8c1-e662-4695-9380-a43b7dd4ced6] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 923b9398-72d1-454e-bc28-1e0bdbf893e1, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 8e17c8c1-e662-4695-9380-a43b7dd4ced6 was re-scheduled: Binding failed for port 923b9398-72d1-454e-bc28-1e0bdbf893e1, please check neutron logs for more information.\n'] [ 997.163994] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 997.164251] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 8e17c8c1-e662-4695-9380-a43b7dd4ced6.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 8e17c8c1-e662-4695-9380-a43b7dd4ced6. [ 997.166202] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: 8e17c8c1-e662-4695-9380-a43b7dd4ced6] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 8e17c8c1-e662-4695-9380-a43b7dd4ced6. [ 997.196781] nova-conductor[54692]: DEBUG nova.network.neutron [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: 8e17c8c1-e662-4695-9380-a43b7dd4ced6] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 997.211412] nova-conductor[54692]: DEBUG nova.network.neutron [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: 8e17c8c1-e662-4695-9380-a43b7dd4ced6] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 997.214353] nova-conductor[54692]: DEBUG nova.network.neutron [None req-73f389c2-ca88-49a0-a802-59cd15dd39ce tempest-ServerDiskConfigTestJSON-215884773 tempest-ServerDiskConfigTestJSON-215884773-project-member] [instance: 8e17c8c1-e662-4695-9380-a43b7dd4ced6] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 999.154052] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:UPDATE=165,nova_cell1:SELECT=334,nova_cell1:INSERT=25,nova_cell1:SAVEPOINT=40,nova_cell1:RELEASE=40 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 1003.122810] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 1003.138844] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 1003.156306] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=4,nova_cell1:UPDATE=3,nova_cell1:INSERT=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 1004.432676] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell0:SELECT=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 1004.451480] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:SELECT=3,nova_cell1:INSERT=7 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 1004.457010] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=7,nova_api:UPDATE=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 1004.460565] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_api:SELECT=5,nova_api:DELETE=1 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 1007.366899] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 5732777e-b491-4e60-bc28-20f3f636f6fe] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port b10809fc-4402-4884-a6c6-a6a4a7484394, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 5732777e-b491-4e60-bc28-20f3f636f6fe was re-scheduled: Binding failed for port b10809fc-4402-4884-a6c6-a6a4a7484394, please check neutron logs for more information.\n'] [ 1007.367572] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 1007.367704] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 5732777e-b491-4e60-bc28-20f3f636f6fe.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 5732777e-b491-4e60-bc28-20f3f636f6fe. [ 1007.367914] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 5732777e-b491-4e60-bc28-20f3f636f6fe] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 5732777e-b491-4e60-bc28-20f3f636f6fe. [ 1007.389146] nova-conductor[54692]: DEBUG nova.network.neutron [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 5732777e-b491-4e60-bc28-20f3f636f6fe] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 1007.403319] nova-conductor[54692]: DEBUG nova.network.neutron [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 5732777e-b491-4e60-bc28-20f3f636f6fe] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 1007.405983] nova-conductor[54692]: DEBUG nova.network.neutron [None req-cf938004-bd7c-419e-8bd3-232528ea955c tempest-DeleteServersTestJSON-57768650 tempest-DeleteServersTestJSON-57768650-project-member] [instance: 5732777e-b491-4e60-bc28-20f3f636f6fe] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 1011.737924] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Expecting reply to msg 41bce4a3ca1748f28ee21b93b8b8702d in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 1011.862678] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 41bce4a3ca1748f28ee21b93b8b8702d [ 1012.247390] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 1012.265932] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 1012.266169] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 1012.266339] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 1012.301819] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 1012.302046] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 1012.302212] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 1012.302668] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 1012.302853] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 1012.303008] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 1012.312146] nova-conductor[54692]: DEBUG nova.quota [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Getting quotas for project af0fcc89cc8548f7b5f9689df14993c4. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 1012.315575] nova-conductor[54692]: DEBUG nova.quota [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Getting quotas for user 88899e5c37af4416a1d5e65b61ca6218 and project af0fcc89cc8548f7b5f9689df14993c4. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 1012.320414] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] [instance: 14bc65e1-9305-4d75-8d27-d3077e7120c9] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 1012.320837] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 1012.321029] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 1012.321190] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 1012.325968] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] [instance: 14bc65e1-9305-4d75-8d27-d3077e7120c9] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 1012.326583] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 1012.326781] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 1012.326968] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 1012.345865] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 1012.346064] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 1012.346228] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 1022.307691] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell0:SELECT=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 1022.328454] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=3,nova_cell1:SAVEPOINT=1,nova_cell1:INSERT=8,nova_cell1:RELEASE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 1022.342283] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=6 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 1022.344340] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=6,nova_api:UPDATE=1,nova_api:DELETE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 1024.910907] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] [instance: 14bc65e1-9305-4d75-8d27-d3077e7120c9] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 67538936-c0fe-4f58-b367-f771cadee631, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 14bc65e1-9305-4d75-8d27-d3077e7120c9 was re-scheduled: Binding failed for port 67538936-c0fe-4f58-b367-f771cadee631, please check neutron logs for more information.\n'] [ 1024.911464] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 1024.911721] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 14bc65e1-9305-4d75-8d27-d3077e7120c9.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 14bc65e1-9305-4d75-8d27-d3077e7120c9. [ 1024.912056] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] [instance: 14bc65e1-9305-4d75-8d27-d3077e7120c9] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 14bc65e1-9305-4d75-8d27-d3077e7120c9. [ 1024.934282] nova-conductor[54692]: DEBUG nova.network.neutron [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] [instance: 14bc65e1-9305-4d75-8d27-d3077e7120c9] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 1024.952885] nova-conductor[54692]: DEBUG nova.network.neutron [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] [instance: 14bc65e1-9305-4d75-8d27-d3077e7120c9] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 1024.956578] nova-conductor[54692]: DEBUG nova.network.neutron [None req-229ecb77-b262-4800-bb68-c7bd04213283 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] [instance: 14bc65e1-9305-4d75-8d27-d3077e7120c9] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 1026.843268] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Expecting reply to msg a7fa409c71c44748954df75e2f163889 in queue reply_be4b6a8028724e64b8a3a40877a630ee [ 1026.958681] nova-conductor[54692]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg a7fa409c71c44748954df75e2f163889 [ 1027.346586] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=54692) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 1027.359433] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 1027.359725] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 1027.359898] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 1027.391624] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 1027.391811] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 1027.391970] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 1027.392346] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 1027.392528] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 1027.392683] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 1027.400470] nova-conductor[54692]: DEBUG nova.quota [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Getting quotas for project af0fcc89cc8548f7b5f9689df14993c4. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 1027.403387] nova-conductor[54692]: DEBUG nova.quota [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Getting quotas for user 88899e5c37af4416a1d5e65b61ca6218 and project af0fcc89cc8548f7b5f9689df14993c4. Resources: {'cores', 'ram', 'instances'} {{(pid=54692) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 1027.407946] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] [instance: abd9a353-18d2-45ac-842f-21c331cf96d8] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=54692) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1761}} [ 1027.408355] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 1027.408545] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 1027.408708] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 1027.413253] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] [instance: abd9a353-18d2-45ac-842f-21c331cf96d8] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='81867c62-ef8e-483f-bfd2-854abdcd6db5',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=54692) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1506}} [ 1027.413839] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 1027.414031] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 1027.414192] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 1027.426221] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Acquiring lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 1027.426416] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 1027.426583] nova-conductor[54692]: DEBUG oslo_concurrency.lockutils [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Lock "2d4e634d-6292-45c4-9eb1-09b106cacc48" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=54692) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 1037.397372] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell0:SELECT=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 1037.416791] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=3,nova_cell1:INSERT=7 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 1037.422897] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=7 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 1037.424864] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_api:SELECT=5,nova_api:UPDATE=1,nova_api:DELETE=1 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 1040.105819] nova-conductor[54692]: ERROR nova.scheduler.utils [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] [instance: abd9a353-18d2-45ac-842f-21c331cf96d8] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port d37d800f-d6c9-456c-95c7-f40794d52f37, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance abd9a353-18d2-45ac-842f-21c331cf96d8 was re-scheduled: Binding failed for port d37d800f-d6c9-456c-95c7-f40794d52f37, please check neutron logs for more information.\n'] [ 1040.106444] nova-conductor[54692]: DEBUG nova.conductor.manager [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Rescheduling: True {{(pid=54692) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 1040.106677] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance abd9a353-18d2-45ac-842f-21c331cf96d8.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance abd9a353-18d2-45ac-842f-21c331cf96d8. [ 1040.106990] nova-conductor[54692]: WARNING nova.scheduler.utils [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] [instance: abd9a353-18d2-45ac-842f-21c331cf96d8] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance abd9a353-18d2-45ac-842f-21c331cf96d8. [ 1040.141836] nova-conductor[54692]: DEBUG nova.network.neutron [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] [instance: abd9a353-18d2-45ac-842f-21c331cf96d8] deallocate_for_instance() {{(pid=54692) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 1040.156415] nova-conductor[54692]: DEBUG nova.network.neutron [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] [instance: abd9a353-18d2-45ac-842f-21c331cf96d8] Instance cache missing network info. {{(pid=54692) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 1040.158948] nova-conductor[54692]: DEBUG nova.network.neutron [None req-59e9656d-9cff-4e73-b73c-886ad4974d48 tempest-AttachVolumeShelveTestJSON-1951917446 tempest-AttachVolumeShelveTestJSON-1951917446-project-member] [instance: abd9a353-18d2-45ac-842f-21c331cf96d8] Updating instance_info_cache with network_info: [] {{(pid=54692) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 1053.305060] nova-conductor[54692]: DEBUG dbcounter [-] [54692] Writing DB stats nova_cell1:SELECT=119,nova_cell1:UPDATE=63,nova_cell1:SAVEPOINT=13,nova_cell1:RELEASE=13,nova_cell1:INSERT=9,nova_cell1:DELETE=2 {{(pid=54692) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 1053.813984] nova-conductor[54691]: DEBUG dbcounter [-] [54691] Writing DB stats nova_cell1:SELECT=92,nova_cell1:UPDATE=48,nova_cell1:SAVEPOINT=9,nova_cell1:RELEASE=9,nova_cell1:INSERT=5 {{(pid=54691) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}}