[ 374.907182] nova-conductor[53267]: INFO dbcounter [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] Registered counter for database nova_api [ 374.916243] nova-conductor[53267]: DEBUG oslo_db.sqlalchemy.engines [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=53267) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:342}} [ 374.944347] nova-conductor[53267]: DEBUG dbcounter [-] [53267] Writer thread running {{(pid=53267) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} [ 374.946458] nova-conductor[53267]: DEBUG nova.context [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] Found 2 cells: 00000000-0000-0000-0000-000000000000(cell0),84364d98-e6ce-480f-8cca-682929b0326f(cell1) {{(pid=53267) load_cells /opt/stack/nova/nova/context.py:464}} [ 374.948122] nova-conductor[53267]: DEBUG oslo_concurrency.lockutils [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53267) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 374.948349] nova-conductor[53267]: DEBUG oslo_concurrency.lockutils [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53267) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 374.948773] nova-conductor[53267]: DEBUG oslo_concurrency.lockutils [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53267) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 374.949177] nova-conductor[53267]: DEBUG oslo_concurrency.lockutils [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53267) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 374.949362] nova-conductor[53267]: DEBUG oslo_concurrency.lockutils [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53267) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 374.950290] nova-conductor[53267]: DEBUG oslo_concurrency.lockutils [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=53267) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 374.951583] nova-conductor[53267]: INFO dbcounter [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] Registered counter for database nova_cell0 [ 374.953928] nova-conductor[53267]: INFO dbcounter [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] Registered counter for database nova_cell1 [ 374.956660] nova-conductor[53267]: DEBUG oslo_db.sqlalchemy.engines [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=53267) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:342}} [ 374.957145] nova-conductor[53267]: DEBUG oslo_db.sqlalchemy.engines [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=53267) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:342}} [ 375.035240] nova-conductor[53267]: DEBUG dbcounter [-] [53267] Writer thread running {{(pid=53267) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} [ 375.036893] nova-conductor[53267]: DEBUG dbcounter [-] [53267] Writer thread running {{(pid=53267) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} [ 375.038903] nova-conductor[53267]: DEBUG oslo_concurrency.lockutils [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] Acquiring lock "singleton_lock" {{(pid=53267) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 375.039072] nova-conductor[53267]: DEBUG oslo_concurrency.lockutils [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] Acquired lock "singleton_lock" {{(pid=53267) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 375.039350] nova-conductor[53267]: DEBUG oslo_concurrency.lockutils [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] Releasing lock "singleton_lock" {{(pid=53267) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 375.039724] nova-conductor[53267]: INFO oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] Starting 2 workers [ 375.044016] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] Started child 53688 {{(pid=53267) _start_child /opt/stack/data/venv/lib/python3.10/site-packages/oslo_service/service.py:579}} [ 375.048022] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] Started child 53689 {{(pid=53267) _start_child /opt/stack/data/venv/lib/python3.10/site-packages/oslo_service/service.py:579}} [ 375.048536] nova-conductor[53688]: INFO nova.service [-] Starting conductor node (version 0.0.1) [ 375.050242] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] Full set of CONF: {{(pid=53267) wait /opt/stack/data/venv/lib/python3.10/site-packages/oslo_service/service.py:653}} [ 375.050438] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ******************************************************************************** {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2600}} [ 375.050597] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] Configuration options gathered from: {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2601}} [ 375.050770] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] command line args: ['--config-file', '/etc/nova/nova.conf'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2602}} [ 375.051111] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] config files: ['/etc/nova/nova.conf'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2603}} [ 375.051266] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ================================================================================ {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2605}} [ 375.051676] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] allow_resize_to_same_host = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.051888] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] arq_binding_timeout = 300 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.052149] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] block_device_allocate_retries = 60 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.052446] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] block_device_allocate_retries_interval = 3 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.052776] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cert = self.pem {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.053173] nova-conductor[53689]: INFO nova.service [-] Starting conductor node (version 0.0.1) [ 375.053412] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] compute_driver = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.053758] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] compute_monitors = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.054112] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] config_dir = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.054458] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] config_drive_format = iso9660 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.054705] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] config_file = ['/etc/nova/nova.conf'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.055021] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] config_source = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.055324] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] console_host = devstack {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.055723] nova-conductor[53688]: INFO dbcounter [None req-4aa0fb9d-d094-488a-b209-ed0546380705 None None] Registered counter for database nova_cell1 [ 375.055951] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] control_exchange = nova {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.056285] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cpu_allocation_ratio = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.056560] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] daemon = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.056868] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] debug = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.057138] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] default_access_ip_network_name = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.057416] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] default_availability_zone = nova {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.057678] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] default_ephemeral_format = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.057949] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] default_green_pool_size = 1000 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.058385] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'glanceclient=WARN', 'oslo.privsep.daemon=INFO'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.058821] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] default_schedule_zone = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.059099] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] disk_allocation_ratio = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.059370] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] enable_new_services = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.059726] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] enabled_apis = ['osapi_compute'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.059856] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] enabled_ssl_apis = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.060046] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] flat_injected = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.060209] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] force_config_drive = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.060503] nova-conductor[53689]: INFO dbcounter [None req-04439138-ca0c-482a-8f20-9ea6302762d4 None None] Registered counter for database nova_cell1 [ 375.060650] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] force_raw_images = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.060848] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] graceful_shutdown_timeout = 5 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.061038] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] heal_instance_info_cache_interval = 60 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.061541] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] host = devstack {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.061784] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] initial_cpu_allocation_ratio = 4.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.062014] nova-conductor[53688]: DEBUG oslo_db.sqlalchemy.engines [None req-4aa0fb9d-d094-488a-b209-ed0546380705 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=53688) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:342}} [ 375.062168] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] initial_disk_allocation_ratio = 1.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.062369] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] initial_ram_allocation_ratio = 1.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.062642] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] injected_network_template = /opt/stack/nova/nova/virt/interfaces.template {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.062835] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] instance_build_timeout = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.063021] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] instance_delete_interval = 300 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.063190] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] instance_format = [instance: %(uuid)s] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.063370] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] instance_name_template = instance-%08x {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.063561] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] instance_usage_audit = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.063770] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] instance_usage_audit_period = month {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.063912] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.064111] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] instances_path = /opt/stack/data/nova/instances {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.064306] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] internal_service_availability_zone = internal {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.064465] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] key = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.064641] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] live_migration_retry_count = 30 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.064803] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] log_color = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.064963] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] log_config_append = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.065142] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.065300] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] log_dir = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.065473] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] log_file = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.065605] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] log_options = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.065790] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] log_rotate_interval = 1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.065978] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] log_rotate_interval_type = days {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.066160] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] log_rotation_type = none {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.066289] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.066493] nova-conductor[53689]: DEBUG oslo_db.sqlalchemy.engines [None req-04439138-ca0c-482a-8f20-9ea6302762d4 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=53689) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:342}} [ 375.066632] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.066819] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.067009] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.067146] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.067343] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] long_rpc_timeout = 1800 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.067499] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] max_concurrent_builds = 10 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.067654] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] max_concurrent_live_migrations = 1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.067832] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] max_concurrent_snapshots = 5 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.067987] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] max_local_block_devices = 3 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.068180] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] max_logfile_count = 30 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.068340] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] max_logfile_size_mb = 200 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.068492] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] maximum_instance_delete_attempts = 5 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.068684] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] metadata_listen = 0.0.0.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.068889] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] metadata_listen_port = 8775 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.069073] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] metadata_workers = 2 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.069253] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] migrate_max_retries = -1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.069421] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] mkisofs_cmd = genisoimage {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.069677] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] my_block_storage_ip = 10.180.1.21 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.069820] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] my_ip = 10.180.1.21 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.069984] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] network_allocate_retries = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.070185] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] non_inheritable_image_properties = ['cache_in_nova', 'bittorrent'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.070351] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] osapi_compute_listen = 0.0.0.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.070515] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] osapi_compute_listen_port = 8774 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.070682] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] osapi_compute_unique_server_name_scope = {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.070882] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] osapi_compute_workers = 2 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.071048] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] password_length = 12 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.071225] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] periodic_enable = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.071393] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] periodic_fuzzy_delay = 60 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.071582] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] pointer_model = usbtablet {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.071792] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] preallocate_images = none {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.071950] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] publish_errors = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.072090] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] pybasedir = /opt/stack/nova {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.072248] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ram_allocation_ratio = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.072428] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] rate_limit_burst = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.072597] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] rate_limit_except_level = CRITICAL {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.072760] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writer thread running {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} [ 375.072903] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] rate_limit_interval = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.073080] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] reboot_timeout = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.073243] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] reclaim_instance_interval = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.073419] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] record = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.073575] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] reimage_timeout_per_gb = 20 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.073746] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] report_interval = 120 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.073902] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] rescue_timeout = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.074054] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] reserved_host_cpus = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.074232] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] reserved_host_disk_mb = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.074396] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] reserved_host_memory_mb = 512 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.074578] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] reserved_huge_pages = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.074738] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] resize_confirm_window = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.074911] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] resize_fs_using_block_device = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.075067] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] resume_guests_state_on_host_boot = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.075229] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] rootwrap_config = /etc/nova/rootwrap.conf {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.075407] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] rpc_response_timeout = 60 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.075569] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] run_external_periodic_tasks = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.075736] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] running_deleted_instance_action = reap {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.075893] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] running_deleted_instance_poll_interval = 1800 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.076084] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] running_deleted_instance_timeout = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.076246] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] scheduler_instance_sync_interval = 120 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.076411] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] service_down_time = 720 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.076572] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writer thread running {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} [ 375.076715] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] servicegroup_driver = db {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.076892] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] shelved_offload_time = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.077054] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] shelved_poll_interval = 3600 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.077224] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] shutdown_timeout = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.077386] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] source_is_ipv6 = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.077554] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ssl_only = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.077737] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] state_path = /opt/stack/data/nova {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.077898] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] sync_power_state_interval = 600 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.078056] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] sync_power_state_pool_size = 1000 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.078216] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] syslog_log_facility = LOG_USER {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.078366] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] tempdir = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.078510] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] timeout_nbd = 10 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.078672] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] transport_url = **** {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.078826] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] update_resources_interval = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.078981] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] use_cow_images = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.079132] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] use_eventlog = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.079310] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] use_journal = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.079484] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] use_json = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.079659] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] use_rootwrap_daemon = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.079815] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] use_stderr = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.079974] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] use_syslog = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.080141] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vcpu_pin_set = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.080308] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vif_plugging_is_fatal = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.080472] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vif_plugging_timeout = 300 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.080666] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] virt_mkfs = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.080828] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] volume_usage_poll_interval = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.080983] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] watch_log_file = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.081164] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] web = /usr/share/spice-html5 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2613}} [ 375.081452] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_concurrency.disable_process_locking = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.081709] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_concurrency.lock_path = /opt/stack/data/nova {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.081932] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.082121] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.082296] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_metrics.metrics_process_name = {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.082465] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.082649] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.082880] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api.auth_strategy = keystone {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.083062] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api.compute_link_prefix = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.083253] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api.config_drive_skip_versions = 1.0 2007-01-19 2007-03-01 2007-08-29 2007-10-10 2007-12-15 2008-02-01 2008-09-01 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.083439] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api.dhcp_domain = novalocal {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.083606] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api.enable_instance_password = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.083771] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api.glance_link_prefix = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.084876] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api.instance_list_cells_batch_fixed_size = 100 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.084876] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api.instance_list_cells_batch_strategy = distributed {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.084876] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api.instance_list_per_project_cells = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.084876] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api.list_records_by_skipping_down_cells = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.089219] nova-conductor[53688]: DEBUG nova.service [None req-4aa0fb9d-d094-488a-b209-ed0546380705 None None] Creating RPC server for service conductor {{(pid=53688) start /opt/stack/nova/nova/service.py:186}} [ 375.089274] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api.local_metadata_per_cell = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.089274] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api.max_limit = 1000 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.089274] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api.metadata_cache_expiration = 15 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.089274] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api.neutron_default_tenant_id = default {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.089274] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api.response_validation = warn {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.089274] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api.use_neutron_default_nets = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.089274] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api.vendordata_dynamic_connect_timeout = 5 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.089578] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api.vendordata_dynamic_failure_fatal = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.089578] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api.vendordata_dynamic_read_timeout = 5 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.089578] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api.vendordata_dynamic_ssl_certfile = {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.089578] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api.vendordata_dynamic_targets = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.089578] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api.vendordata_jsonfile_path = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.089578] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api.vendordata_providers = ['StaticJSON'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.089578] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.backend = dogpile.cache.memcached {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.089890] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.backend_argument = **** {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.089890] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.config_prefix = cache.oslo {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.089890] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.dead_timeout = 60.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.089890] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.debug_cache_backend = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.089890] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.enable_retry_client = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.089890] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.enable_socket_keepalive = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.089890] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.enabled = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.090189] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.enforce_fips_mode = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.090189] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.expiration_time = 600 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.090189] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.hashclient_retry_attempts = 2 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.090189] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.hashclient_retry_delay = 1.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.090189] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.memcache_dead_retry = 300 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.090189] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.memcache_password = **** {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.090189] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.memcache_pool_connection_get_timeout = 10 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.090504] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.memcache_pool_flush_on_reconnect = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.090504] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.memcache_pool_maxsize = 10 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.090504] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.memcache_pool_unused_timeout = 60 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.090504] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.memcache_sasl_enabled = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.090687] nova-conductor[53689]: DEBUG nova.service [None req-04439138-ca0c-482a-8f20-9ea6302762d4 None None] Creating RPC server for service conductor {{(pid=53689) start /opt/stack/nova/nova/service.py:186}} [ 375.090737] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.memcache_servers = ['localhost:11211'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.090737] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.memcache_socket_timeout = 1.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.090737] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.memcache_username = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.090909] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.proxies = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.091096] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.redis_db = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.091230] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.redis_password = **** {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.091393] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.redis_sentinel_service_name = mymaster {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.091605] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.redis_sentinels = ['localhost:26379'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.091786] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.redis_server = localhost:6379 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.091960] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.redis_socket_timeout = 1.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.092144] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.redis_username = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.092305] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.retry_attempts = 2 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.092467] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.retry_delay = 0.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.092624] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.socket_keepalive_count = 1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.092779] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.socket_keepalive_idle = 1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.092935] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.socket_keepalive_interval = 1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.093109] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.tls_allowed_ciphers = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.093262] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.tls_cafile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.093415] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.tls_certfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.093574] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.tls_enabled = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.093747] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cache.tls_keyfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.093969] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cinder.auth_section = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.094146] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cinder.auth_type = password {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.094310] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cinder.cafile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.094501] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cinder.catalog_info = volumev3::publicURL {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.094660] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cinder.certfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.094821] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cinder.collect_timing = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.094979] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cinder.cross_az_attach = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.095136] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cinder.debug = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.095316] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cinder.endpoint_template = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.095493] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cinder.http_retries = 3 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.095659] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cinder.insecure = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.095815] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cinder.keyfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.096015] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cinder.os_region_name = RegionOne {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.096184] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cinder.split_loggers = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.096350] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cinder.timeout = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.096513] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] compute.consecutive_build_service_disable_threshold = 10 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.096670] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] compute.cpu_dedicated_set = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.096855] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] compute.cpu_shared_set = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.097022] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] compute.image_type_exclude_list = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.097180] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] compute.live_migration_wait_for_vif_plug = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.097341] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] compute.max_concurrent_disk_ops = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.097505] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] compute.max_disk_devices_to_attach = -1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.097665] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] compute.packing_host_numa_cells_allocation_strategy = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.098237] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] compute.provider_config_location = /etc/nova/provider_config/ {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.098592] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] compute.resource_provider_association_refresh = 300 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.098793] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] compute.sharing_providers_max_uuids_per_request = 200 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.098964] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] compute.shutdown_retry_interval = 10 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.099147] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] compute.vmdk_allowed_types = ['streamOptimized', 'monolithicSparse'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.100599] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] conductor.workers = 2 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.100809] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] console.allowed_origins = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.100982] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] console.ssl_ciphers = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.101160] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] console.ssl_minimum_version = default {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.101401] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] consoleauth.enforce_session_timeout = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.103978] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] consoleauth.token_ttl = 600 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.106980] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cyborg.cafile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.107698] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cyborg.certfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.108032] nova-conductor[53689]: INFO oslo.messaging._drivers.impl_rabbit [None req-04439138-ca0c-482a-8f20-9ea6302762d4 None None] Creating fanout queue: conductor_fanout_2af5e494d0e54eb7b007f8f93d449ade [ 375.108187] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cyborg.collect_timing = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.108750] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cyborg.connect_retries = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.109114] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cyborg.connect_retry_delay = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.109440] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cyborg.endpoint_override = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.109670] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cyborg.insecure = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.109841] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cyborg.keyfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.110354] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cyborg.max_version = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.110641] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cyborg.min_version = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.110866] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cyborg.region_name = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.111055] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cyborg.retriable_status_codes = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.111415] nova-conductor[53689]: DEBUG nova.service [None req-04439138-ca0c-482a-8f20-9ea6302762d4 None None] Join ServiceGroup membership for this service conductor {{(pid=53689) start /opt/stack/nova/nova/service.py:203}} [ 375.111592] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cyborg.service_name = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.111880] nova-conductor[53689]: DEBUG nova.servicegroup.drivers.db [None req-04439138-ca0c-482a-8f20-9ea6302762d4 None None] DB_Driver: join new ServiceGroup member devstack to the conductor group, service = {{(pid=53689) join /opt/stack/nova/nova/servicegroup/drivers/db.py:44}} [ 375.112038] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cyborg.service_type = accelerator {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.112236] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cyborg.split_loggers = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.112708] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cyborg.status_code_retries = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.112945] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cyborg.status_code_retry_delay = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.113125] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cyborg.timeout = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.113316] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cyborg.valid_interfaces = ['internal', 'public'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.113478] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] cyborg.version = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.113675] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] database.backend = sqlalchemy {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.113848] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] database.connection = **** {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.114012] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] database.connection_debug = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.114183] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] database.connection_parameters = {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.114343] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] database.connection_recycle_time = 3600 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.114505] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] database.connection_trace = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.114697] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] database.db_inc_retry_interval = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.114878] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] database.db_max_retries = 20 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.115046] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] database.db_max_retry_interval = 10 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.115205] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] database.db_retry_interval = 1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.115365] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] database.max_overflow = 50 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.115526] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] database.max_pool_size = 5 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.115683] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] database.max_retries = 10 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.115856] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] database.mysql_sql_mode = TRADITIONAL {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.116050] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] database.mysql_wsrep_sync_wait = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.116188] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] database.pool_timeout = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.116349] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] database.retry_interval = 10 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.116503] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] database.slave_connection = **** {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.116667] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] database.sqlite_synchronous = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.117079] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] database.use_db_reconnect = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.117408] nova-conductor[53688]: INFO oslo.messaging._drivers.impl_rabbit [None req-4aa0fb9d-d094-488a-b209-ed0546380705 None None] Creating fanout queue: conductor_fanout_e61ea5a7878344f2876dbe7d3af4540a [ 375.117552] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api_database.backend = sqlalchemy {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.117940] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api_database.connection = **** {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.118162] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api_database.connection_debug = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.118566] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api_database.connection_parameters = {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.119643] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api_database.connection_recycle_time = 3600 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.119951] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api_database.connection_trace = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.120159] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api_database.db_inc_retry_interval = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.120652] nova-conductor[53688]: DEBUG nova.service [None req-4aa0fb9d-d094-488a-b209-ed0546380705 None None] Join ServiceGroup membership for this service conductor {{(pid=53688) start /opt/stack/nova/nova/service.py:203}} [ 375.120814] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api_database.db_max_retries = 20 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.121114] nova-conductor[53688]: DEBUG nova.servicegroup.drivers.db [None req-4aa0fb9d-d094-488a-b209-ed0546380705 None None] DB_Driver: join new ServiceGroup member devstack to the conductor group, service = {{(pid=53688) join /opt/stack/nova/nova/servicegroup/drivers/db.py:44}} [ 375.121274] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api_database.db_max_retry_interval = 10 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.121497] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api_database.db_retry_interval = 1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.121894] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api_database.max_overflow = 50 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.122077] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api_database.max_pool_size = 5 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.122244] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api_database.max_retries = 10 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.122414] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api_database.mysql_sql_mode = TRADITIONAL {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.122577] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api_database.mysql_wsrep_sync_wait = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.122734] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api_database.pool_timeout = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.122893] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api_database.retry_interval = 10 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.123046] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api_database.slave_connection = **** {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.123204] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] api_database.sqlite_synchronous = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.123448] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] devices.enabled_mdev_types = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.123642] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ephemeral_storage_encryption.cipher = aes-xts-plain64 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.123814] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ephemeral_storage_encryption.default_format = luks {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.123976] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ephemeral_storage_encryption.enabled = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.124155] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ephemeral_storage_encryption.key_size = 512 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.124359] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.api_servers = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.124524] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.cafile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.124700] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.certfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.124861] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.collect_timing = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.125030] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.connect_retries = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.125187] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.connect_retry_delay = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.125362] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.debug = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.125539] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.default_trusted_certificate_ids = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.125700] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.enable_certificate_validation = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.125858] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.enable_rbd_download = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.126031] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.endpoint_override = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.126196] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.insecure = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.126375] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.keyfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.126540] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.max_version = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.126697] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.min_version = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.126853] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.num_retries = 3 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.127015] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.rbd_ceph_conf = {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.127177] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.rbd_connect_timeout = 5 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.127342] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.rbd_pool = {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.127514] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.rbd_user = {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.127690] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.region_name = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.127848] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.retriable_status_codes = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.128009] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.service_name = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.128195] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.service_type = image {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.128354] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.split_loggers = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.128508] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.status_code_retries = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.128666] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.status_code_retry_delay = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.128817] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.timeout = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.128991] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.valid_interfaces = ['internal', 'public'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.129151] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.verify_glance_signatures = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.129318] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] glance.version = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.129534] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] guestfs.debug = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.129719] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] mks.enabled = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.130308] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] mks.mksproxy_base_url = http://127.0.0.1:6090/ {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.130525] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] image_cache.manager_interval = 2400 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.130712] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] image_cache.precache_concurrency = 1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.130884] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] image_cache.remove_unused_base_images = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.131050] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] image_cache.remove_unused_original_minimum_age_seconds = 86400 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.131215] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] image_cache.remove_unused_resized_minimum_age_seconds = 3600 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.131391] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] image_cache.subdirectory_name = _base {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.131615] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.api_max_retries = 60 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.131844] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.api_retry_interval = 2 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.132112] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.auth_section = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.132223] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.auth_type = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.132378] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.cafile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.132558] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.certfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.132730] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.collect_timing = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.132915] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.conductor_group = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.133076] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.connect_retries = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.133234] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.connect_retry_delay = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.133388] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.endpoint_override = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.133547] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.insecure = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.133704] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.keyfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.133856] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.max_version = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.134009] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.min_version = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.134173] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.peer_list = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.134329] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.region_name = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.134489] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.retriable_status_codes = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.134647] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.serial_console_state_timeout = 10 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.134832] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.service_name = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.135026] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.service_type = baremetal {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.135183] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.shard = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.135347] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.split_loggers = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.135506] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.status_code_retries = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.135663] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.status_code_retry_delay = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.135818] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.timeout = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.135997] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.valid_interfaces = ['internal', 'public'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.136183] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ironic.version = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.136384] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] key_manager.backend = nova.keymgr.conf_key_mgr.ConfKeyManager {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.136572] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] key_manager.fixed_key = **** {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.136789] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican.auth_endpoint = http://localhost/identity/v3 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.136963] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican.barbican_api_version = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.137118] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican.barbican_endpoint = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.137314] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican.barbican_endpoint_type = public {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.137488] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican.barbican_region_name = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.137647] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican.cafile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.137860] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican.certfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.138037] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican.collect_timing = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.138216] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican.insecure = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.138377] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican.keyfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.138537] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican.number_of_retries = 60 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.138700] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican.retry_delay = 1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.138863] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican.send_service_user_token = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.139020] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican.split_loggers = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.139174] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican.timeout = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.139330] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican.verify_ssl = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.139511] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican.verify_ssl_path = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.139718] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican_service_user.auth_section = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.139882] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican_service_user.auth_type = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.140050] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican_service_user.cafile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.140207] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican_service_user.certfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.140375] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican_service_user.collect_timing = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.140532] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican_service_user.insecure = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.140686] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican_service_user.keyfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.140878] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican_service_user.split_loggers = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.141037] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] barbican_service_user.timeout = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.141223] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vault.approle_role_id = **** {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.141384] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vault.approle_secret_id = **** {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.141573] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vault.kv_mountpoint = secret {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.141736] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vault.kv_path = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.141893] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vault.kv_version = 2 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.142051] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vault.namespace = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.142205] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vault.root_token_id = **** {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.142358] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vault.ssl_ca_crt_file = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.142539] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vault.timeout = 60.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.142705] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vault.use_ssl = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.142890] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vault.vault_url = http://127.0.0.1:8200 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.143080] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] keystone.auth_section = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.143241] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] keystone.auth_type = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.143394] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] keystone.cafile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.143570] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] keystone.certfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.143741] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] keystone.collect_timing = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.143919] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] keystone.connect_retries = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.144094] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] keystone.connect_retry_delay = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.144252] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] keystone.endpoint_override = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.144409] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] keystone.insecure = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.144562] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] keystone.keyfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.144733] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] keystone.max_version = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.144906] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] keystone.min_version = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.145062] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] keystone.region_name = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.145221] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] keystone.retriable_status_codes = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.145375] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] keystone.service_name = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.145561] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] keystone.service_type = identity {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.145722] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] keystone.split_loggers = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.145876] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] keystone.status_code_retries = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.146033] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] keystone.status_code_retry_delay = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.146186] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] keystone.timeout = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.146360] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] keystone.valid_interfaces = ['internal', 'public'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.146516] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] keystone.version = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.146817] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.connection_uri = {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.147033] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.cpu_mode = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.147250] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.cpu_model_extra_flags = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.147423] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.cpu_models = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.147592] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.cpu_power_governor_high = performance {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.147759] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.cpu_power_governor_low = powersave {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.147919] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.cpu_power_management = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.148123] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.cpu_power_management_strategy = cpu_state {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.148306] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.device_detach_attempts = 8 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.148490] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.device_detach_timeout = 20 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.148655] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.disk_cachemodes = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.148811] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.disk_prefix = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.148976] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.enabled_perf_events = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.149133] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.file_backed_memory = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.149295] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.gid_maps = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.149483] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.hw_disk_discard = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.149658] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.hw_machine_type = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.149827] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.images_rbd_ceph_conf = {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.149988] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.images_rbd_glance_copy_poll_interval = 15 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.150151] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.images_rbd_glance_copy_timeout = 600 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.150317] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.images_rbd_glance_store_name = {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.150480] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.images_rbd_pool = rbd {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.150669] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.images_type = default {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.150828] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.images_volume_group = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.151010] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.inject_key = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.151170] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.inject_partition = -2 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.151328] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.inject_password = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.151509] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.iscsi_iface = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.151707] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.iser_use_multipath = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.151871] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.live_migration_bandwidth = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.152044] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.live_migration_completion_timeout = 800 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.152207] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.live_migration_downtime = 500 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.152368] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.live_migration_downtime_delay = 75 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.152547] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.live_migration_downtime_steps = 10 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.152744] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.live_migration_inbound_addr = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.152907] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.live_migration_permit_auto_converge = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.153066] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.live_migration_permit_post_copy = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.153221] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.live_migration_scheme = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.153389] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.live_migration_timeout_action = abort {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.153549] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.live_migration_tunnelled = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.153706] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.live_migration_uri = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.153864] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.live_migration_with_native_tls = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.154043] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.max_queues = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.154204] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.mem_stats_period_seconds = 10 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.154451] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.migration_inbound_addr = 10.180.1.21 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.154616] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.nfs_mount_options = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.154841] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.nfs_mount_point_base = /opt/stack/data/nova/mnt {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.155007] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.num_aoe_discover_tries = 3 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.155187] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.num_iser_scan_tries = 5 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.155357] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.num_memory_encrypted_guests = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.155558] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.num_nvme_discover_tries = 5 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.155745] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.num_pcie_ports = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.155911] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.num_volume_scan_tries = 5 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.156109] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.pmem_namespaces = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.156279] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.quobyte_client_cfg = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.156501] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.quobyte_mount_point_base = /opt/stack/data/nova/mnt {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.156665] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.rbd_connect_timeout = 5 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.156828] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.rbd_destroy_volume_retries = 12 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.156988] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.rbd_destroy_volume_retry_interval = 5 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.157144] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.rbd_secret_uuid = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.157295] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.rbd_user = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.157452] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.realtime_scheduler_priority = 1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.157639] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.remote_filesystem_transport = ssh {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.157796] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.rescue_image_id = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.157946] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.rescue_kernel_id = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.158098] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.rescue_ramdisk_id = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.158260] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.rng_dev_path = /dev/urandom {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.158415] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.rx_queue_size = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.158600] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.smbfs_mount_options = {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.158827] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.smbfs_mount_point_base = /opt/stack/data/nova/mnt {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.158989] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.snapshot_compression = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.159147] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.snapshot_image_format = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.159355] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.snapshots_directory = /opt/stack/data/nova/instances/snapshots {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.159519] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.sparse_logical_volumes = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.159679] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.swtpm_enabled = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.159864] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.swtpm_group = tss {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.160043] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.swtpm_user = tss {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.160216] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.sysinfo_serial = unique {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.160372] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.tb_cache_size = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.160529] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.tx_queue_size = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.160691] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.uid_maps = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.160847] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.use_virtio_for_bridges = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.161036] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.virt_type = kvm {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.161199] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.volume_clear = zero {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.161357] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.volume_clear_size = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.161552] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.volume_use_multipath = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.161738] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.vzstorage_cache_path = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.161908] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.vzstorage_log_path = /var/log/vstorage/%(cluster_name)s/nova.log.gz {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.162073] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.vzstorage_mount_group = qemu {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.162238] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.vzstorage_mount_opts = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.162402] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.vzstorage_mount_perms = 0770 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.162637] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.vzstorage_mount_point_base = /opt/stack/data/nova/mnt {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.162811] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.vzstorage_mount_user = stack {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.162974] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] libvirt.wait_soft_reboot_seconds = 120 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.163161] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.auth_section = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.163347] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.auth_type = password {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.163501] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.cafile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.163655] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.certfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.163813] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.collect_timing = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.163965] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.connect_retries = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.164163] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.connect_retry_delay = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.164365] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.default_floating_pool = public {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.164524] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.endpoint_override = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.164716] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.extension_sync_interval = 600 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.164884] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.http_retries = 3 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.165073] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.insecure = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.165235] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.keyfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.165391] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.max_version = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.165563] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.metadata_proxy_shared_secret = **** {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.165719] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.min_version = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.165880] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.ovs_bridge = br-int {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.166467] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.physnets = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.166644] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.region_name = RegionOne {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.166803] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.retriable_status_codes = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.166971] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.service_metadata_proxy = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.167128] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.service_name = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.167293] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.service_type = network {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.167453] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.split_loggers = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.167620] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.status_code_retries = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.167834] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.status_code_retry_delay = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.167997] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.timeout = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.168241] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.valid_interfaces = ['internal', 'public'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.168373] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] neutron.version = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.168544] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] notifications.bdms_in_notifications = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.168742] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] notifications.default_level = INFO {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.168917] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] notifications.notification_format = unversioned {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.169078] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] notifications.notify_on_state_change = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.169248] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] notifications.versioned_notifications_topics = ['versioned_notifications'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.169420] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] pci.alias = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.169587] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] pci.device_spec = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.169749] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] pci.report_in_placement = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.169961] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.auth_section = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.170131] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.auth_type = password {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.170299] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.auth_url = http://10.180.1.21/identity {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.170456] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.cafile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.170611] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.certfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.170799] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.collect_timing = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.170963] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.connect_retries = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.171121] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.connect_retry_delay = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.171275] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.default_domain_id = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.171428] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.default_domain_name = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.171606] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.domain_id = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.171767] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.domain_name = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.171923] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.endpoint_override = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.172092] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.insecure = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.172247] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.keyfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.172399] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.max_version = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.172552] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.min_version = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.172716] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.password = **** {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.172894] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.project_domain_id = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.173058] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.project_domain_name = Default {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.173223] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.project_id = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.173391] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.project_name = service {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.173555] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.region_name = RegionOne {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.173741] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.retriable_status_codes = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.173915] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.service_name = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.174083] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.service_type = placement {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.174242] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.split_loggers = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.174398] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.status_code_retries = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.174552] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.status_code_retry_delay = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.174706] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.system_scope = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.174858] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.timeout = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.175011] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.trust_id = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.175164] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.user_domain_id = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.175330] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.user_domain_name = Default {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.175486] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.user_id = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.175652] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.username = nova {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.175829] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.valid_interfaces = ['internal', 'public'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.175983] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] placement.version = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.176197] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] quota.cores = 20 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.176367] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] quota.count_usage_from_placement = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.176534] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] quota.driver = nova.quota.DbQuotaDriver {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.176740] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] quota.injected_file_content_bytes = 10240 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.176924] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] quota.injected_file_path_length = 255 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.177088] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] quota.injected_files = 5 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.177251] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] quota.instances = 10 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.177412] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] quota.key_pairs = 100 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.177574] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] quota.metadata_items = 128 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.177737] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] quota.ram = 51200 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.177896] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] quota.recheck_quota = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.178056] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] quota.server_group_members = 10 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.178217] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] quota.server_groups = 10 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.178422] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] scheduler.discover_hosts_in_cells_interval = -1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.178605] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] scheduler.enable_isolated_aggregate_filtering = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.178786] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] scheduler.image_metadata_prefilter = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.178944] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] scheduler.limit_tenants_to_placement_aggregate = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.179111] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] scheduler.max_attempts = 3 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.179273] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] scheduler.max_placement_results = 1000 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.179470] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] scheduler.placement_aggregate_required_for_tenants = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.179651] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] scheduler.query_placement_for_image_type_support = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.179836] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] scheduler.query_placement_for_routed_network_aggregates = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.180014] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] scheduler.workers = 2 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.180228] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.aggregate_image_properties_isolation_namespace = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.180400] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.aggregate_image_properties_isolation_separator = . {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.180594] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.available_filters = ['nova.scheduler.filters.all_filters'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.180762] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.build_failure_weight_multiplier = 1000000.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.180924] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.cpu_weight_multiplier = 1.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.181114] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.cross_cell_move_weight_multiplier = 1000000.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.181295] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.disk_weight_multiplier = 1.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.181530] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.enabled_filters = ['ComputeFilter', 'ComputeCapabilitiesFilter', 'ImagePropertiesFilter', 'ServerGroupAntiAffinityFilter', 'ServerGroupAffinityFilter', 'SameHostFilter', 'DifferentHostFilter'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.181720] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.host_subset_size = 1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.181885] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.hypervisor_version_weight_multiplier = 1.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.182061] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.image_properties_default_architecture = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.182228] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.io_ops_weight_multiplier = -1.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.182390] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.isolated_hosts = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.182591] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.isolated_images = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.182767] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.max_instances_per_host = 50 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.182949] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.max_io_ops_per_host = 8 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.183110] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.num_instances_weight_multiplier = 0.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.183273] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.pci_in_placement = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.183433] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.pci_weight_multiplier = 1.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.183590] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.ram_weight_multiplier = 1.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.183748] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.restrict_isolated_hosts_to_isolated_images = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.183903] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.shuffle_best_same_weighed_hosts = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.184086] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.soft_affinity_weight_multiplier = 1.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.184263] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.soft_anti_affinity_weight_multiplier = 1.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.184424] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.track_instance_changes = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.184596] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] filter_scheduler.weight_classes = ['nova.scheduler.weights.all_weighers'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.184763] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] metrics.required = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.184919] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] metrics.weight_multiplier = 1.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.185079] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] metrics.weight_of_unavailable = -10000.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.185240] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] metrics.weight_setting = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.185574] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] serial_console.base_url = ws://127.0.0.1:6083/ {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.185763] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] serial_console.enabled = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.185959] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] serial_console.port_range = 10000:20000 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.186130] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] serial_console.proxyclient_address = 127.0.0.1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.186295] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] serial_console.serialproxy_host = 0.0.0.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.186460] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] serial_console.serialproxy_port = 6083 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.186662] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] service_user.auth_section = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.186880] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] service_user.auth_type = password {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.187059] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] service_user.cafile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.187217] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] service_user.certfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.187379] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] service_user.collect_timing = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.187535] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] service_user.insecure = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.187688] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] service_user.keyfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.187855] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] service_user.send_service_user_token = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.188028] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] service_user.split_loggers = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.188205] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] service_user.timeout = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.188395] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] spice.agent_enabled = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.188575] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] spice.enabled = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.188929] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] spice.html5proxy_base_url = http://127.0.0.1:6082/spice_auto.html {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.189133] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] spice.html5proxy_host = 0.0.0.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.189304] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] spice.html5proxy_port = 6082 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.189492] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] spice.image_compression = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.189651] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] spice.jpeg_compression = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.189808] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] spice.playback_compression = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.189975] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] spice.server_listen = 127.0.0.1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.190138] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] spice.server_proxyclient_address = 127.0.0.1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.190295] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] spice.streaming_mode = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.190446] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] spice.zlib_compression = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.190645] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] upgrade_levels.baseapi = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.190840] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] upgrade_levels.compute = auto {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.191003] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] upgrade_levels.conductor = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.191167] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] upgrade_levels.scheduler = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.191337] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vendordata_dynamic_auth.auth_section = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.191534] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vendordata_dynamic_auth.auth_type = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.191718] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vendordata_dynamic_auth.cafile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.191880] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vendordata_dynamic_auth.certfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.192053] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vendordata_dynamic_auth.collect_timing = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.192219] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vendordata_dynamic_auth.insecure = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.192375] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vendordata_dynamic_auth.keyfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.192547] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vendordata_dynamic_auth.split_loggers = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.192695] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vendordata_dynamic_auth.timeout = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.192911] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.api_retry_count = 10 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.193072] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.ca_file = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.193227] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.cache_prefix = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.193377] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.cluster_name = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.193533] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.connection_pool_size = 10 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.193685] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.console_delay_seconds = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.193836] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.datastore_regex = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.193991] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.host_ip = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.194144] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.host_password = **** {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.194305] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.host_port = 443 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.194465] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.host_username = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.194638] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.insecure = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.194811] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.integration_bridge = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.194969] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.maximum_objects = 100 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.195122] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.pbm_default_policy = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.195278] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.pbm_enabled = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.195435] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.pbm_wsdl_location = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.195598] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.serial_log_dir = /opt/vmware/vspc {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.195758] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.serial_port_proxy_uri = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.195913] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.serial_port_service_uri = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.196095] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.task_poll_interval = 0.5 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.196260] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.use_linked_clone = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.196426] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.vnc_keymap = en-us {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.196585] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.vnc_port = 5900 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.196746] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vmware.vnc_port_total = 10000 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.196960] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vnc.auth_schemes = ['none'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.197126] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vnc.enabled = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.197436] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vnc.novncproxy_base_url = http://127.0.0.1:6080/vnc_auto.html {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.197630] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vnc.novncproxy_host = 0.0.0.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.197820] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vnc.novncproxy_port = 6080 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.198001] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vnc.server_listen = 127.0.0.1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.198169] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vnc.server_proxyclient_address = 127.0.0.1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.198328] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vnc.vencrypt_ca_certs = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.198484] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vnc.vencrypt_client_cert = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.198644] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] vnc.vencrypt_client_key = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.198849] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] workarounds.disable_compute_service_check_for_ffu = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.199010] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] workarounds.disable_deep_image_inspection = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.199166] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] workarounds.disable_fallback_pcpu_query = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.199319] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] workarounds.disable_group_policy_check_upcall = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.199481] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] workarounds.disable_libvirt_livesnapshot = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.199633] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] workarounds.disable_rootwrap = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.199790] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] workarounds.enable_numa_live_migration = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.199946] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] workarounds.enable_qemu_monitor_announce_self = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.200118] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] workarounds.ensure_libvirt_rbd_instance_dir_cleanup = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.200277] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] workarounds.handle_virt_lifecycle_events = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.200434] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] workarounds.libvirt_disable_apic = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.200589] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] workarounds.never_download_image_if_on_rbd = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.200803] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] workarounds.qemu_monitor_announce_self_count = 3 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.200987] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] workarounds.qemu_monitor_announce_self_interval = 1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.201148] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] workarounds.reserve_disk_resource_for_image_cache = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.201307] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] workarounds.skip_cpu_compare_at_startup = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.201482] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] workarounds.skip_cpu_compare_on_dest = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.201662] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] workarounds.skip_hypervisor_version_check_on_lm = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.201827] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] workarounds.skip_reserve_in_use_ironic_nodes = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.201988] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] workarounds.unified_limits_count_pcpu_as_vcpu = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.202149] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] workarounds.wait_for_vif_plugged_event_during_hard_reboot = [] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.202348] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] wsgi.api_paste_config = /etc/nova/api-paste.ini {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.202514] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] wsgi.client_socket_timeout = 900 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.202677] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] wsgi.default_pool_size = 1000 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.202835] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] wsgi.keep_alive = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.202994] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] wsgi.max_header_line = 16384 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.203151] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] wsgi.secure_proxy_ssl_header = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.203304] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] wsgi.ssl_ca_file = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.203457] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] wsgi.ssl_cert_file = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.203615] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] wsgi.ssl_key_file = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.203991] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] wsgi.tcp_keepidle = 600 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.204226] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] wsgi.wsgi_log_format = %(client_ip)s "%(request_line)s" status: %(status_code)s len: %(body_length)s time: %(wall_seconds).7f {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.204398] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] zvm.ca_file = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.204933] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] zvm.cloud_connector_url = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.205199] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] zvm.image_tmp_path = /opt/stack/data/nova/images {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.205374] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] zvm.reachable_timeout = 300 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.205610] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_policy.enforce_new_defaults = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.205789] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_policy.enforce_scope = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.205981] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_policy.policy_default_rule = default {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.206168] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.206357] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_policy.policy_file = policy.yaml {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.206568] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.206750] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.206924] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.207083] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.207239] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.207442] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.207649] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.207848] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] profiler.connection_string = messaging:// {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.208166] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] profiler.enabled = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.208391] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] profiler.es_doc_type = notification {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.208574] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] profiler.es_scroll_size = 10000 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.208745] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] profiler.es_scroll_time = 2m {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.209039] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] profiler.filter_error_trace = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.209222] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] profiler.hmac_keys = **** {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.209386] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] profiler.sentinel_service_name = mymaster {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.209569] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] profiler.socket_timeout = 0.1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.209737] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] profiler.trace_requests = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.209893] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] profiler.trace_sqlalchemy = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.210097] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] profiler_jaeger.process_tags = {} {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.210257] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] profiler_jaeger.service_name_prefix = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.210436] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] profiler_otlp.service_name_prefix = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.210628] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] remote_debug.host = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.210792] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] remote_debug.port = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.210982] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.211145] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.211309] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.211496] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.211672] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.211834] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.211999] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.heartbeat_in_pthread = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.212172] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.heartbeat_rate = 3 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.212329] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.212516] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.hostname = devstack {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.212687] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.212861] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.213025] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.213196] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.213367] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.processname = nova-conductor {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.213532] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.213694] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.213882] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.214052] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.214208] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.rabbit_quorum_delivery_limit = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.214367] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.rabbit_quorum_max_memory_bytes = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.214527] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.rabbit_quorum_max_memory_length = 0 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.214688] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.214850] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.215008] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.215166] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.rabbit_stream_fanout = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.215330] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.215488] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.rabbit_transient_quorum_queue = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.215687] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.215863] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.ssl = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.216046] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.216220] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.216379] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.ssl_enforce_fips_mode = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.216544] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.216709] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.ssl_version = {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.216865] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_rabbit.use_queue_manager = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.217090] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_notifications.driver = ['messagingv2'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.217252] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_notifications.retry = -1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.217449] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.217621] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_messaging_notifications.transport_url = **** {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.217821] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_limit.auth_section = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.217984] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_limit.auth_type = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.218139] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_limit.cafile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.218290] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_limit.certfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.218452] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_limit.collect_timing = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.218637] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_limit.connect_retries = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.218820] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_limit.connect_retry_delay = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.218979] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_limit.endpoint_id = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.219132] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_limit.endpoint_override = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.219286] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_limit.insecure = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.219438] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_limit.keyfile = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.219588] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_limit.max_version = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.219740] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_limit.min_version = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.219890] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_limit.region_name = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.220056] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_limit.retriable_status_codes = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.220218] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_limit.service_name = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.220371] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_limit.service_type = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.220526] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_limit.split_loggers = False {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.220680] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_limit.status_code_retries = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.220837] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_limit.status_code_retry_delay = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.220987] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_limit.timeout = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.221140] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_limit.valid_interfaces = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.221289] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_limit.version = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.221511] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_reports.file_event_handler = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.221721] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_reports.file_event_handler_interval = 1 {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.221885] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] oslo_reports.log_dir = None {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2620}} [ 375.222014] nova-conductor[53267]: DEBUG oslo_service.service [None req-8a6bb764-df17-4eb1-9a5c-4951c7a5d6cc None None] ******************************************************************************** {{(pid=53267) log_opt_values /opt/stack/data/venv/lib/python3.10/site-packages/oslo_config/cfg.py:2624}} [ 384.946428] nova-conductor[53267]: DEBUG dbcounter [-] [53267] Writing DB stats nova_api:SELECT=1 {{(pid=53267) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 385.037318] nova-conductor[53267]: DEBUG dbcounter [-] [53267] Writing DB stats nova_cell1:SELECT=1 {{(pid=53267) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 385.038127] nova-conductor[53267]: DEBUG dbcounter [-] [53267] Writing DB stats nova_cell0:SELECT=1 {{(pid=53267) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 390.122342] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=5,nova_cell1:INSERT=1,nova_cell1:UPDATE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 390.127655] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SELECT=4,nova_cell1:INSERT=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 467.887306] nova-conductor[53688]: DEBUG oslo_db.sqlalchemy.engines [None req-b8416d59-acfc-47a3-b747-d8892a506a78 None None] Parent process 53267 forked (53688) with an open database connection, which is being discarded and recreated. {{(pid=53688) checkout /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:438}} [ 467.889475] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writer thread running {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} [ 477.890403] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 485.010715] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=7,nova_cell1:INSERT=1,nova_cell1:UPDATE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 490.529660] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SELECT=7,nova_cell1:INSERT=1,nova_cell1:UPDATE=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 510.118658] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=1,nova_cell1:UPDATE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 510.123619] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SELECT=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 515.349579] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writer thread running {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} [ 515.357443] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Creating reply queue: reply_5133ae70522a4309b9b1926cdadcc99f [ 515.368104] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Expecting reply to msg 736c1f4168864763a152f854a080818b in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 515.773349] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 736c1f4168864763a152f854a080818b [ 515.877232] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Took 0.52 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 515.890542] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=2 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 515.910479] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 515.910785] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 515.913584] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.002s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 515.916645] nova-conductor[53689]: INFO dbcounter [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Registered counter for database nova_cell1 [ 515.922443] nova-conductor[53689]: DEBUG oslo_db.sqlalchemy.engines [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=53689) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:342}} [ 515.928196] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writer thread running {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} [ 516.010564] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 516.010791] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 516.011256] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 516.011687] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 516.011875] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 516.012040] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 516.013630] nova-conductor[53689]: INFO dbcounter [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Registered counter for database nova_cell0 [ 516.021856] nova-conductor[53689]: DEBUG oslo_db.sqlalchemy.engines [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=53689) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:342}} [ 516.027858] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writer thread running {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} [ 516.039814] nova-conductor[53689]: DEBUG nova.quota [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Getting quotas for project c06b50a7aaa742afbbd0c6fc56c3d131. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 516.043232] nova-conductor[53689]: DEBUG nova.quota [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Getting quotas for user 3078a2af81b248f8b100f58ee66a5a86 and project c06b50a7aaa742afbbd0c6fc56c3d131. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 516.051677] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 52eb3ce9-696c-445c-b82d-90663a9e8b21] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 516.052250] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 516.052452] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 516.052602] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 516.061777] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 52eb3ce9-696c-445c-b82d-90663a9e8b21] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 516.062443] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 516.062645] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 516.062805] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 516.104883] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 516.105125] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 516.105292] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 516.105559] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "compute-rpcapi-router" {{(pid=53689) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 516.105692] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquired lock "compute-rpcapi-router" {{(pid=53689) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 516.106236] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e4e2ccd5-f7b6-4768-9b2f-f2ecabeb60c6 None None] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 516.106426] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e4e2ccd5-f7b6-4768-9b2f-f2ecabeb60c6 None None] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 516.106901] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e4e2ccd5-f7b6-4768-9b2f-f2ecabeb60c6 None None] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 516.107395] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e4e2ccd5-f7b6-4768-9b2f-f2ecabeb60c6 None None] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 516.107812] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e4e2ccd5-f7b6-4768-9b2f-f2ecabeb60c6 None None] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 516.107984] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e4e2ccd5-f7b6-4768-9b2f-f2ecabeb60c6 None None] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 516.118528] nova-conductor[53689]: INFO nova.compute.rpcapi [None req-e4e2ccd5-f7b6-4768-9b2f-f2ecabeb60c6 None None] Automatically selected compute RPC version 6.3 from minimum service version 67 [ 516.119708] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e4e2ccd5-f7b6-4768-9b2f-f2ecabeb60c6 None None] Releasing lock "compute-rpcapi-router" {{(pid=53689) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 516.178013] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Expecting reply to msg 00a81a6ce5c14079b9a116a4d1eb7e80 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 516.297523] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Creating reply queue: reply_d19a0b27ff1248c0876daa8ff85cc326 [ 516.304510] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Expecting reply to msg 126ec43cb8564bfe9bc3d38ea83c5f01 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 516.581983] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 00a81a6ce5c14079b9a116a4d1eb7e80 [ 516.586170] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Expecting reply to msg 7c1b4b2a5c4744f7be4959ea6dac6383 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 516.678621] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 126ec43cb8564bfe9bc3d38ea83c5f01 [ 516.681274] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 516.709192] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 516.709424] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 516.709629] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 516.812253] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Took 0.52 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 516.815870] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=2 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 516.848736] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 516.849016] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 516.850710] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.002s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 516.852695] nova-conductor[53688]: INFO dbcounter [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Registered counter for database nova_cell1 [ 516.863064] nova-conductor[53688]: DEBUG oslo_db.sqlalchemy.engines [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=53688) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:342}} [ 516.873749] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writer thread running {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} [ 516.896468] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 516.896468] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 516.896468] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 516.896468] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 516.896929] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 516.896929] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 516.916539] nova-conductor[53689]: DEBUG nova.quota [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Getting quotas for project 3b11f5c3d754416d81b53d294f3f9631. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 516.922124] nova-conductor[53689]: DEBUG nova.quota [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Getting quotas for user 0c66923fec264dcf9cf04011a1463650 and project 3b11f5c3d754416d81b53d294f3f9631. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 516.942547] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] [instance: 986f91ac-e44f-474a-885a-bc097b396019] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 516.943355] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 516.943658] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 516.943917] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 516.963557] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 7c1b4b2a5c4744f7be4959ea6dac6383 [ 516.966868] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] [instance: 986f91ac-e44f-474a-885a-bc097b396019] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 516.969038] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 516.969038] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 516.969038] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 516.971756] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 516.973483] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.002s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 516.974064] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 516.974598] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 516.974898] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 516.975184] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 516.976719] nova-conductor[53688]: INFO dbcounter [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Registered counter for database nova_cell0 [ 516.993194] nova-conductor[53688]: DEBUG oslo_db.sqlalchemy.engines [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=53688) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:342}} [ 517.008348] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writer thread running {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} [ 517.022012] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 517.022721] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 517.022721] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 517.024866] nova-conductor[53688]: DEBUG nova.quota [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Getting quotas for project 088552f0960945fab5e28d10e79cb726. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 517.030264] nova-conductor[53688]: DEBUG nova.quota [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Getting quotas for user a8fa21bcd84747c0bca54261bdc441c4 and project 088552f0960945fab5e28d10e79cb726. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 517.041327] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] [instance: 020a5b3a-bda7-4a8a-9dad-948cee5a7373] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 517.041996] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 517.042140] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 517.042303] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 517.051238] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] [instance: 020a5b3a-bda7-4a8a-9dad-948cee5a7373] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 517.052063] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 517.052231] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 517.052414] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 517.093442] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 517.094538] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 517.094743] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 517.095011] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 517.095166] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Acquiring lock "compute-rpcapi-router" {{(pid=53688) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:310}} [ 517.095306] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Acquired lock "compute-rpcapi-router" {{(pid=53688) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:313}} [ 517.095854] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8d605505-0c1e-44ed-9f48-c01459f51921 None None] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 517.096030] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8d605505-0c1e-44ed-9f48-c01459f51921 None None] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 517.096379] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8d605505-0c1e-44ed-9f48-c01459f51921 None None] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 517.096551] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8d605505-0c1e-44ed-9f48-c01459f51921 None None] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 517.096715] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8d605505-0c1e-44ed-9f48-c01459f51921 None None] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 517.096841] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8d605505-0c1e-44ed-9f48-c01459f51921 None None] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 517.112121] nova-conductor[53688]: INFO nova.compute.rpcapi [None req-8d605505-0c1e-44ed-9f48-c01459f51921 None None] Automatically selected compute RPC version 6.3 from minimum service version 67 [ 517.112707] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8d605505-0c1e-44ed-9f48-c01459f51921 None None] Releasing lock "compute-rpcapi-router" {{(pid=53688) lock /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:331}} [ 517.114598] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 517.114843] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 517.115016] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 517.199623] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 517.200349] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 517.200861] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 517.201483] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 517.201896] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 517.202270] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 517.223716] nova-conductor[53689]: DEBUG nova.quota [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Getting quotas for project e50b6ecfe84a44d6821b987722cfa474. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 517.241650] nova-conductor[53689]: DEBUG nova.quota [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Getting quotas for user c082680fa2254a3286797922adf42920 and project e50b6ecfe84a44d6821b987722cfa474. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 517.312593] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] [instance: 4d75a1ec-6c11-4cdd-ba0e-aa6d3add80f2] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 517.313279] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 517.313491] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 517.313664] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 517.320546] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] [instance: 4d75a1ec-6c11-4cdd-ba0e-aa6d3add80f2] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 517.321358] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 517.321616] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 517.321880] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 517.351813] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 517.352125] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 517.352382] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a67fda18-1c60-4ab4-89ee-00087cc2ef3a tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 517.776402] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Expecting reply to msg 5faf03deb4024b51863979977a1e070a in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 517.855973] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Expecting reply to msg f4333af1de06417fba74cfa3ae2ed6e7 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 517.947456] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 5faf03deb4024b51863979977a1e070a [ 518.008416] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg f4333af1de06417fba74cfa3ae2ed6e7 [ 518.279655] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 518.294874] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 518.295111] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 518.295287] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 518.351263] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 518.351674] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 518.351722] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 518.352082] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 518.352277] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 518.352437] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 518.360340] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 518.370385] nova-conductor[53688]: DEBUG nova.quota [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Getting quotas for project 9562b8ed4a894a4da10f11866133a628. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 518.372785] nova-conductor[53688]: DEBUG nova.quota [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Getting quotas for user af469e3bdb504baeaa90377cd88815cf and project 9562b8ed4a894a4da10f11866133a628. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 518.383865] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] [instance: 8f2cba43-bdec-4455-b795-784b29e2ea5d] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 518.384410] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 518.384719] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 518.384930] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 518.389392] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 518.389677] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 518.389848] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 518.396191] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] [instance: 8f2cba43-bdec-4455-b795-784b29e2ea5d] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 518.396874] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 518.397076] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 518.397236] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 518.431779] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 518.432145] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 518.432192] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 518.439577] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 518.439766] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 518.439906] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 518.440354] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 518.440548] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 518.440708] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 518.450633] nova-conductor[53689]: DEBUG nova.quota [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Getting quotas for project 67c0ae43fbdd470dae9b8035b1c8022e. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 518.453191] nova-conductor[53689]: DEBUG nova.quota [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Getting quotas for user e3159b2750904f0caf5cbd42cf43a8c9 and project 67c0ae43fbdd470dae9b8035b1c8022e. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 518.457888] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] [instance: ea496eae-68f9-43f1-b4cf-6743043c753b] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 518.458377] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 518.458585] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 518.458746] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 518.462176] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] [instance: ea496eae-68f9-43f1-b4cf-6743043c753b] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 518.463010] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 518.463236] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 518.463402] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 518.476630] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 518.476832] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 518.476992] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 518.832188] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Expecting reply to msg 7a94d1f34c7f4f3ca5339856e2a5c620 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 519.007966] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 7a94d1f34c7f4f3ca5339856e2a5c620 [ 519.336030] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 519.349788] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 519.350085] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 519.350325] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 519.382196] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 519.382629] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 519.382948] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 519.383433] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 519.383738] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 519.383999] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 519.394850] nova-conductor[53688]: DEBUG nova.quota [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Getting quotas for project 2e2cc0c992514f71a9718faad2e13951. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 519.398727] nova-conductor[53688]: DEBUG nova.quota [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Getting quotas for user 1cd76e5588fa4d49bd07b2276708cbb9 and project 2e2cc0c992514f71a9718faad2e13951. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 519.406364] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] [instance: f58fbc9b-ba60-4c56-bbdf-fbaf45ec12b5] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 519.406873] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 519.407074] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 519.407244] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 519.412721] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] [instance: f58fbc9b-ba60-4c56-bbdf-fbaf45ec12b5] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 519.413431] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 519.413633] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 519.413833] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 519.435106] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 519.435337] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 519.435503] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 521.292714] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Expecting reply to msg 6e5b7f3baa2544e5ac34c34f2cc64b3a in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 521.461911] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 6e5b7f3baa2544e5ac34c34f2cc64b3a [ 521.798468] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 521.823984] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 521.824252] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 521.824425] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 521.862637] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 521.862885] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 521.863174] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 521.863424] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 521.863610] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 521.863776] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 521.873202] nova-conductor[53688]: DEBUG nova.quota [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Getting quotas for project 2e2cc0c992514f71a9718faad2e13951. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 521.883092] nova-conductor[53688]: DEBUG nova.quota [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Getting quotas for user 1cd76e5588fa4d49bd07b2276708cbb9 and project 2e2cc0c992514f71a9718faad2e13951. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 521.888463] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] [instance: 40709d2b-cc33-4ac0-9a13-731442d7edff] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 521.889124] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 521.889274] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 521.889440] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 521.894906] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] [instance: 40709d2b-cc33-4ac0-9a13-731442d7edff] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 521.895578] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 521.895778] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 521.895945] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 521.917655] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 521.918683] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 521.918683] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 522.940979] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Expecting reply to msg d7b9fdad18a04dd288980a2286c365ec in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 523.098805] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg d7b9fdad18a04dd288980a2286c365ec [ 523.443970] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 523.460860] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 523.461091] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 523.461169] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 523.518376] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 523.518664] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 523.518876] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 523.519719] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 523.519719] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 523.519719] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 523.534644] nova-conductor[53689]: DEBUG nova.quota [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Getting quotas for project 25a75c216e55475b8b8e05870de35b29. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 523.537859] nova-conductor[53689]: DEBUG nova.quota [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Getting quotas for user 206eaa30e3aa4fa982e48dd99c20f44a and project 25a75c216e55475b8b8e05870de35b29. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 523.551299] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] [instance: 1784917b-8a7e-4974-b8b3-f8f2b3db019a] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 523.551887] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 523.552193] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 523.552401] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 523.557294] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] [instance: 1784917b-8a7e-4974-b8b3-f8f2b3db019a] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 523.561725] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 523.561725] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 523.561725] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 523.582101] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 523.582290] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 523.582459] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 526.964095] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Expecting reply to msg bcf1f975982249c686b345379f24db86 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 527.155641] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg bcf1f975982249c686b345379f24db86 [ 527.467654] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 527.484788] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 527.485026] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 527.485231] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 527.537891] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 527.538140] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 527.538323] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 527.538705] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 527.538880] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 527.539041] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 527.553550] nova-conductor[53689]: DEBUG nova.quota [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Getting quotas for project 67f289a0c54e46e18b97b47c11a9b1f5. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 527.557465] nova-conductor[53689]: DEBUG nova.quota [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Getting quotas for user 6116fae5f64e41e09dd7d6f6fcab88dd and project 67f289a0c54e46e18b97b47c11a9b1f5. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 527.563330] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] [instance: 228821ca-e981-405b-8952-8a1718103d3c] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 527.563916] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 527.564945] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 527.564945] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 527.569796] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] [instance: 228821ca-e981-405b-8952-8a1718103d3c] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 527.572750] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 527.573466] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 527.573466] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 527.603429] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 527.603719] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 527.604096] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 528.225130] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] Expecting reply to msg ace5a59c1a8640dc92c225d4e382c2df in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 528.401043] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg ace5a59c1a8640dc92c225d4e382c2df [ 528.730750] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 528.773195] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 528.773435] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 528.773607] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 528.819051] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 528.819201] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 528.819383] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 528.819771] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 528.819962] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 528.820160] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 528.839921] nova-conductor[53688]: DEBUG nova.quota [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] Getting quotas for project 921b38590177446c941b2872e63cf976. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 528.844359] nova-conductor[53688]: DEBUG nova.quota [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] Getting quotas for user 3ba7aefab5d64de198502dec5ecabe49 and project 921b38590177446c941b2872e63cf976. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 528.858611] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] [instance: 62c3b24d-bee7-4dd2-a6c7-9303c7c28cca] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 528.859722] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 528.860244] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 528.860420] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 528.866921] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] [instance: 62c3b24d-bee7-4dd2-a6c7-9303c7c28cca] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 528.868288] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 528.868456] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 528.868919] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 528.898827] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 528.899204] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 528.899495] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4b20b822-c6f9-46a9-9852-aa9f838979ee tempest-ServerDiagnosticsV248Test-943853116 tempest-ServerDiagnosticsV248Test-943853116-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 529.080073] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Expecting reply to msg 31132f2ed3dc4318b51800982c41c53e in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 529.306487] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 31132f2ed3dc4318b51800982c41c53e [ 529.588283] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 529.634394] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 529.634394] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 529.634394] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 529.798685] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 529.798685] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 529.798685] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 529.798685] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 529.798895] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 529.798895] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 529.828760] nova-conductor[53689]: DEBUG nova.quota [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Getting quotas for project 81bbad196c3b40d6b138d3366da6248b. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 529.838641] nova-conductor[53689]: DEBUG nova.quota [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Getting quotas for user 8efc2fd425c047a1bbfa9e05794f23b6 and project 81bbad196c3b40d6b138d3366da6248b. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 529.849556] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] [instance: 1d64d913-45f0-4768-8375-7863d9ae43c3] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 529.850092] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 529.850294] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 529.850455] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 529.867913] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] [instance: 1d64d913-45f0-4768-8375-7863d9ae43c3] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 529.868693] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 529.868901] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 529.869063] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 529.916375] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 529.916619] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 529.916833] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 534.606303] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Expecting reply to msg 3e16242ba2c24401aa08e7f2c8226ed4 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 534.804826] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 3e16242ba2c24401aa08e7f2c8226ed4 [ 535.111685] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 535.134209] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 535.134209] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 535.134209] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 535.204203] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 535.204451] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 535.204641] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 535.204997] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 535.205170] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 535.205322] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 535.218223] nova-conductor[53688]: DEBUG nova.quota [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Getting quotas for project 760605cb896a4cbdb566a044b1e14fdc. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 535.222065] nova-conductor[53688]: DEBUG nova.quota [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Getting quotas for user 2eca29707860426b983cf69143b5d6b4 and project 760605cb896a4cbdb566a044b1e14fdc. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 535.228705] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] [instance: f0c26eb3-e6d6-4d9f-9f07-5add9de6d126] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 535.229279] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 535.229549] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 535.230034] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 535.235529] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] [instance: f0c26eb3-e6d6-4d9f-9f07-5add9de6d126] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 535.236311] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 535.236391] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 535.236553] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 535.256501] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 535.256807] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 535.256985] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 535.984282] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Expecting reply to msg 1ddcf6a3805c48628c435c5f759a1177 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 536.125289] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 1ddcf6a3805c48628c435c5f759a1177 [ 536.487264] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 536.512230] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 536.512652] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 536.512847] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 536.561152] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 536.562759] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.002s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 536.563072] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 536.563529] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 536.563913] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 536.564155] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 536.585477] nova-conductor[53688]: DEBUG nova.quota [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Getting quotas for project ca33c1550e694c38a15f9be28d24b19c. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 536.589230] nova-conductor[53688]: DEBUG nova.quota [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Getting quotas for user 0986847129324981ba5df690cc76bcba and project ca33c1550e694c38a15f9be28d24b19c. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 536.601769] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] [instance: 8f37dfb3-c4d2-4c41-91eb-e3c7ca640050] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 536.601970] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 536.602200] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 536.602393] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 536.608193] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] [instance: 8f37dfb3-c4d2-4c41-91eb-e3c7ca640050] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 536.609024] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 536.609230] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 536.609387] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 536.647127] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 536.647391] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 536.647588] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 537.503097] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Expecting reply to msg 56c401dba64b41d298b1daad00e88874 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 537.737256] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 56c401dba64b41d298b1daad00e88874 [ 538.009105] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 538.027528] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 538.027771] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 538.027952] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 538.079744] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 538.079991] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 538.080168] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 538.080969] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 538.083591] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 538.083875] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.003s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 538.099063] nova-conductor[53689]: DEBUG nova.quota [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Getting quotas for project 2eeaa82f12ce4d39a8d6b32e7cbbcd9f. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 538.101989] nova-conductor[53689]: DEBUG nova.quota [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Getting quotas for user ceaca1b950d44ea7a0e720301e2f707c and project 2eeaa82f12ce4d39a8d6b32e7cbbcd9f. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 538.116984] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] [instance: 3f1f549f-8034-4685-b6f0-db5a7a2a4a32] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 538.117494] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 538.117775] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 538.118084] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 538.128775] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] [instance: 3f1f549f-8034-4685-b6f0-db5a7a2a4a32] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 538.129688] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 538.129925] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 538.130117] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 538.150808] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 538.151532] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 538.151790] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 538.164414] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Expecting reply to msg 9e673cde4ac2482eb15ea7041b29e72a in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 538.312044] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 9e673cde4ac2482eb15ea7041b29e72a [ 538.668800] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 538.810353] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 538.810768] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 538.810819] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 538.930517] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 538.930796] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 538.931009] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 538.931430] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 538.931676] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 538.932159] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 539.008960] nova-conductor[53689]: DEBUG nova.quota [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Getting quotas for project 8973425b51ed44779cd2caf61cddfae3. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 539.015406] nova-conductor[53689]: DEBUG nova.quota [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Getting quotas for user 9fa51619ae0443588ddcffcc0dd9c9e6 and project 8973425b51ed44779cd2caf61cddfae3. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 539.028394] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] [instance: 36c3ac75-5bfd-4a89-9ddb-28fded8da39c] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 539.029006] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 539.029440] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 539.029666] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 539.044902] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] [instance: 36c3ac75-5bfd-4a89-9ddb-28fded8da39c] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 539.045698] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 539.045934] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 539.046111] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 539.083739] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 539.083986] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 539.084343] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 540.029872] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-bbba9f36-2feb-438a-a94b-e06b9012af64 tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Targeting cell 84364d98-e6ce-480f-8cca-682929b0326f(cell1) for conductor method rebuild_instance {{(pid=53689) wrapper /opt/stack/nova/nova/conductor/manager.py:95}} [ 540.030320] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bbba9f36-2feb-438a-a94b-e06b9012af64 tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 540.030634] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bbba9f36-2feb-438a-a94b-e06b9012af64 tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 540.030899] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bbba9f36-2feb-438a-a94b-e06b9012af64 tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 540.059990] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-bbba9f36-2feb-438a-a94b-e06b9012af64 tempest-ServersAdmin275Test-2057105880 tempest-ServersAdmin275Test-2057105880-project-member] [instance: 4d75a1ec-6c11-4cdd-ba0e-aa6d3add80f2] No migration record for the rebuild/evacuate request. {{(pid=53689) rebuild_instance /opt/stack/nova/nova/conductor/manager.py:1227}} [ 540.130862] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Expecting reply to msg f0f33b9f0b91410b90890fb566db12de in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 540.344744] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg f0f33b9f0b91410b90890fb566db12de [ 540.633865] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 540.647415] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 540.647587] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 540.647824] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 540.742684] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 540.742684] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 540.742684] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 540.742894] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 540.743115] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 540.743310] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 540.763968] nova-conductor[53688]: DEBUG nova.quota [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Getting quotas for project 6c9ed91d497142ddb16a51df48b5c996. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 540.768373] nova-conductor[53688]: DEBUG nova.quota [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Getting quotas for user 0bb003baa0764f52a66a6a7253669732 and project 6c9ed91d497142ddb16a51df48b5c996. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 540.776190] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] [instance: 5cddeea1-7558-4c12-afdc-2ea7a706881a] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 540.777644] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 540.777644] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 540.777644] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 540.880455] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] [instance: 5cddeea1-7558-4c12-afdc-2ea7a706881a] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 540.881093] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 540.881202] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 540.881395] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 540.901935] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 52eb3ce9-696c-445c-b82d-90663a9e8b21] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port d6dd597b-ff4a-4e48-b10a-96073004f3eb, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 52eb3ce9-696c-445c-b82d-90663a9e8b21 was re-scheduled: Binding failed for port d6dd597b-ff4a-4e48-b10a-96073004f3eb, please check neutron logs for more information.\n'] [ 540.902711] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 540.903506] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 52eb3ce9-696c-445c-b82d-90663a9e8b21.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 52eb3ce9-696c-445c-b82d-90663a9e8b21. [ 540.903506] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 52eb3ce9-696c-445c-b82d-90663a9e8b21] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 52eb3ce9-696c-445c-b82d-90663a9e8b21. [ 540.985604] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 540.985894] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 540.986106] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 541.028984] nova-conductor[53688]: DEBUG nova.network.neutron [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 52eb3ce9-696c-445c-b82d-90663a9e8b21] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 541.476524] nova-conductor[53688]: DEBUG nova.network.neutron [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 52eb3ce9-696c-445c-b82d-90663a9e8b21] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 541.481644] nova-conductor[53688]: DEBUG nova.network.neutron [None req-e9aaa151-a991-4861-aeec-17e30c5d48e5 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 52eb3ce9-696c-445c-b82d-90663a9e8b21] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 542.471778] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Expecting reply to msg fab25124bb204374ac6caa5e54e1a99c in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 542.656388] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg fab25124bb204374ac6caa5e54e1a99c [ 542.978760] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 542.996880] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 542.997263] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 542.997621] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 543.075501] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 543.075501] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 543.075501] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 543.075501] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 543.075501] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 543.075501] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 543.093375] nova-conductor[53689]: DEBUG nova.quota [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Getting quotas for project 6c9ed91d497142ddb16a51df48b5c996. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 543.097445] nova-conductor[53689]: DEBUG nova.quota [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Getting quotas for user 0bb003baa0764f52a66a6a7253669732 and project 6c9ed91d497142ddb16a51df48b5c996. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 543.110692] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] [instance: 7f8b08d4-3535-48ab-ba3f-a159511e2a64] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 543.110692] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 543.111630] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 543.111874] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 543.127153] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] [instance: 7f8b08d4-3535-48ab-ba3f-a159511e2a64] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 543.134997] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 543.135721] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.002s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 543.135721] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 543.158249] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 543.158455] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 543.158627] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 544.574860] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Expecting reply to msg 5f5695cf4b3d483daf010924a93183b1 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 544.761757] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 5f5695cf4b3d483daf010924a93183b1 [ 545.077418] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 545.093489] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 545.093865] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 545.094082] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 545.141363] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 545.141604] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 545.141780] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 545.142448] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 545.142655] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 545.142849] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 545.160896] nova-conductor[53689]: DEBUG nova.quota [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Getting quotas for project 3de9d2e84f5f44b785cadcfe5adcd3da. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 545.172046] nova-conductor[53689]: DEBUG nova.quota [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Getting quotas for user 77dead37e5de41e99014b84af587c7db and project 3de9d2e84f5f44b785cadcfe5adcd3da. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 545.180587] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] [instance: 7d1f7566-8e5e-476c-9d19-49ed7b16c308] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 545.181111] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 545.181327] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 545.181493] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 545.193700] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] [instance: 7d1f7566-8e5e-476c-9d19-49ed7b16c308] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 545.194483] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 545.194668] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 545.194844] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 545.245936] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 545.245936] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 545.245936] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 545.492995] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] [instance: 986f91ac-e44f-474a-885a-bc097b396019] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 397126f4-dc7b-428e-9725-ba5c0ded1af3, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 986f91ac-e44f-474a-885a-bc097b396019 was re-scheduled: Binding failed for port 397126f4-dc7b-428e-9725-ba5c0ded1af3, please check neutron logs for more information.\n'] [ 545.494184] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 545.494184] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 986f91ac-e44f-474a-885a-bc097b396019.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 986f91ac-e44f-474a-885a-bc097b396019. [ 545.494489] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] [instance: 986f91ac-e44f-474a-885a-bc097b396019] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 986f91ac-e44f-474a-885a-bc097b396019. [ 545.558746] nova-conductor[53689]: DEBUG nova.network.neutron [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] [instance: 986f91ac-e44f-474a-885a-bc097b396019] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 545.678055] nova-conductor[53689]: DEBUG nova.network.neutron [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] [instance: 986f91ac-e44f-474a-885a-bc097b396019] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 545.688918] nova-conductor[53689]: DEBUG nova.network.neutron [None req-41238e4a-b036-49d6-8907-007255bd521e tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] [instance: 986f91ac-e44f-474a-885a-bc097b396019] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 546.279833] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Expecting reply to msg bef72d6ff9c7488cb103d1e628e45e53 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 546.499434] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg bef72d6ff9c7488cb103d1e628e45e53 [ 546.790183] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Took 0.52 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 546.810383] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 546.811031] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 546.811031] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 546.862096] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 546.862391] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 546.862601] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 546.863036] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 546.863256] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 546.863447] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 546.881444] nova-conductor[53689]: DEBUG nova.quota [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Getting quotas for project a3de421e0f994df8b809ce0096753f23. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 546.885667] nova-conductor[53689]: DEBUG nova.quota [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Getting quotas for user c471664da5894985bf7478057ea19b73 and project a3de421e0f994df8b809ce0096753f23. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 546.892143] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: 1d84bcbd-9b55-4d6f-b6c7-24391c8600a7] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 546.892759] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 546.893144] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 546.893218] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 546.899918] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: 1d84bcbd-9b55-4d6f-b6c7-24391c8600a7] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 546.900842] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 546.901194] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 546.901394] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 546.927747] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 546.928165] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 546.928878] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 547.825105] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Expecting reply to msg 520083c147c1492d849fa02adefafaaf in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 548.005513] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 520083c147c1492d849fa02adefafaaf [ 548.327208] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 548.339570] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 548.339833] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 548.339998] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 548.373952] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 548.374218] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 548.374425] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 548.374820] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 548.375012] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 548.375172] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 548.389961] nova-conductor[53688]: DEBUG nova.quota [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Getting quotas for project 0aadc74e25e84da28c962ea370b4abb7. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 548.392852] nova-conductor[53688]: DEBUG nova.quota [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Getting quotas for user e40bbb7692934bf6b18281dd86f8f130 and project 0aadc74e25e84da28c962ea370b4abb7. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 548.397905] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] [instance: 1804f229-97b9-4ee3-933d-715431a900f8] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 548.398566] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 548.398726] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 548.398850] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 548.404345] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] [instance: 1804f229-97b9-4ee3-933d-715431a900f8] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 548.405271] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 548.405485] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 548.405649] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 548.422644] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 548.422898] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 548.423197] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 548.531900] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Expecting reply to msg 93f565b1eb9d4630901e478e93e8cf97 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 548.706102] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 93f565b1eb9d4630901e478e93e8cf97 [ 549.036153] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 549.053717] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 549.053971] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 549.054302] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 549.099746] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 549.100188] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 549.100401] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 549.100896] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 549.101135] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 549.101321] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 549.115286] nova-conductor[53688]: DEBUG nova.quota [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Getting quotas for project c06b50a7aaa742afbbd0c6fc56c3d131. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 549.118997] nova-conductor[53688]: DEBUG nova.quota [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Getting quotas for user 3078a2af81b248f8b100f58ee66a5a86 and project c06b50a7aaa742afbbd0c6fc56c3d131. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 549.127432] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 017b1da4-7c9b-477d-92a3-29b2248317d3] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 549.127432] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 549.127432] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 549.127698] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 549.134587] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 017b1da4-7c9b-477d-92a3-29b2248317d3] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 549.135500] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 549.135804] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 549.135985] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 549.170158] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 549.170501] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 549.170751] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 549.671023] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Expecting reply to msg 7307dccc7e2d4a9a84ed80d58484a7ab in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 549.832157] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 7307dccc7e2d4a9a84ed80d58484a7ab [ 550.032616] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] [instance: 020a5b3a-bda7-4a8a-9dad-948cee5a7373] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 4e2bb2c8-5529-4001-9487-43fbe76c2658, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 020a5b3a-bda7-4a8a-9dad-948cee5a7373 was re-scheduled: Binding failed for port 4e2bb2c8-5529-4001-9487-43fbe76c2658, please check neutron logs for more information.\n'] [ 550.033175] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 550.033175] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 020a5b3a-bda7-4a8a-9dad-948cee5a7373.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 020a5b3a-bda7-4a8a-9dad-948cee5a7373. [ 550.039693] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] [instance: 020a5b3a-bda7-4a8a-9dad-948cee5a7373] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 020a5b3a-bda7-4a8a-9dad-948cee5a7373. [ 550.110715] nova-conductor[53689]: DEBUG nova.network.neutron [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] [instance: 020a5b3a-bda7-4a8a-9dad-948cee5a7373] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 550.167182] nova-conductor[53689]: DEBUG nova.network.neutron [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] [instance: 020a5b3a-bda7-4a8a-9dad-948cee5a7373] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 550.180139] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 550.183881] nova-conductor[53689]: DEBUG nova.network.neutron [None req-71c04b5d-a508-4ca5-84a2-1b16a0081f0d tempest-ServerDiagnosticsNegativeTest-495393484 tempest-ServerDiagnosticsNegativeTest-495393484-project-member] [instance: 020a5b3a-bda7-4a8a-9dad-948cee5a7373] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 550.212920] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 550.213140] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 550.213302] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 550.263030] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 550.263193] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 550.263462] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 550.263836] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 550.263924] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 550.264170] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 550.282224] nova-conductor[53689]: DEBUG nova.quota [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Getting quotas for project 49494417788d472c93cc089b87e577dc. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 550.288032] nova-conductor[53689]: DEBUG nova.quota [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Getting quotas for user 181d2abbc63840d3b1e6b51ec2c6ee0d and project 49494417788d472c93cc089b87e577dc. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 550.295849] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] [instance: 32b57d1b-d35f-488e-be23-9119f2f56562] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 550.296360] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 550.296542] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 550.296733] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 550.299382] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] [instance: 32b57d1b-d35f-488e-be23-9119f2f56562] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 550.300421] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 550.300550] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 550.300679] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 550.317771] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 550.317983] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 550.318146] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 551.893300] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] [instance: 8f2cba43-bdec-4455-b795-784b29e2ea5d] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 86deca19-3809-4ccf-a3db-0d79d610d6c5, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 8f2cba43-bdec-4455-b795-784b29e2ea5d was re-scheduled: Binding failed for port 86deca19-3809-4ccf-a3db-0d79d610d6c5, please check neutron logs for more information.\n'] [ 551.893943] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 551.894188] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 8f2cba43-bdec-4455-b795-784b29e2ea5d.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 8f2cba43-bdec-4455-b795-784b29e2ea5d. [ 551.894514] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] [instance: 8f2cba43-bdec-4455-b795-784b29e2ea5d] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 8f2cba43-bdec-4455-b795-784b29e2ea5d. [ 551.941094] nova-conductor[53688]: DEBUG nova.network.neutron [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] [instance: 8f2cba43-bdec-4455-b795-784b29e2ea5d] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 551.983540] nova-conductor[53688]: DEBUG nova.network.neutron [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] [instance: 8f2cba43-bdec-4455-b795-784b29e2ea5d] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 551.988238] nova-conductor[53688]: DEBUG nova.network.neutron [None req-0043e483-94cf-44cf-8fc5-41e87170cd2b tempest-ImagesNegativeTestJSON-984867956 tempest-ImagesNegativeTestJSON-984867956-project-member] [instance: 8f2cba43-bdec-4455-b795-784b29e2ea5d] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 552.017941] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Expecting reply to msg 1216369284af43729c1f3f10dbb1342c in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 552.181554] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 1216369284af43729c1f3f10dbb1342c [ 552.520735] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 552.534891] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 552.534891] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 552.534891] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 552.591124] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 552.591367] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 552.591626] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 552.592015] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 552.592208] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 552.592371] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 552.600277] nova-conductor[53689]: DEBUG nova.quota [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Getting quotas for project 3b11f5c3d754416d81b53d294f3f9631. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 552.602413] nova-conductor[53689]: DEBUG nova.quota [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Getting quotas for user 0c66923fec264dcf9cf04011a1463650 and project 3b11f5c3d754416d81b53d294f3f9631. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 552.607312] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] [instance: d4d3db12-8de6-4daf-a087-89bb043d1217] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 552.607781] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 552.607982] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 552.608175] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 552.612173] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] [instance: d4d3db12-8de6-4daf-a087-89bb043d1217] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 552.612891] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 552.613176] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 552.613329] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 552.634783] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 552.634971] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 552.635877] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 553.074780] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Expecting reply to msg 323ad921e5d24d218e34bb19b36ce0a9 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 553.224978] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 323ad921e5d24d218e34bb19b36ce0a9 [ 553.577694] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 553.592287] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 553.592539] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 553.592738] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 553.660353] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 553.660353] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 553.660353] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 553.660353] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 553.660353] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 553.660353] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 553.673308] nova-conductor[53688]: DEBUG nova.quota [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Getting quotas for project d6c3b64a422f4697a7bcd3a33a414842. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 553.678005] nova-conductor[53688]: DEBUG nova.quota [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Getting quotas for user e0e2ea2c33c641b6a0a8114ae60f9bfc and project d6c3b64a422f4697a7bcd3a33a414842. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 553.689114] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] [instance: 13c1d417-4087-46ad-b513-fc3317995d18] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 553.689786] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 553.690232] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 553.690545] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 553.695694] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] [instance: 13c1d417-4087-46ad-b513-fc3317995d18] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 553.696648] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 553.696983] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 553.697305] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 553.726263] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 553.726656] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 553.726936] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 553.791195] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] [instance: ea496eae-68f9-43f1-b4cf-6743043c753b] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 2a66073a-39ef-4960-8443-ca3964c62be0, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance ea496eae-68f9-43f1-b4cf-6743043c753b was re-scheduled: Binding failed for port 2a66073a-39ef-4960-8443-ca3964c62be0, please check neutron logs for more information.\n'] [ 553.791857] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 553.792100] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ea496eae-68f9-43f1-b4cf-6743043c753b.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ea496eae-68f9-43f1-b4cf-6743043c753b. [ 553.792907] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] [instance: ea496eae-68f9-43f1-b4cf-6743043c753b] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ea496eae-68f9-43f1-b4cf-6743043c753b. [ 553.833243] nova-conductor[53689]: DEBUG nova.network.neutron [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] [instance: ea496eae-68f9-43f1-b4cf-6743043c753b] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 553.852449] nova-conductor[53689]: DEBUG nova.network.neutron [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] [instance: ea496eae-68f9-43f1-b4cf-6743043c753b] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 553.860152] nova-conductor[53689]: DEBUG nova.network.neutron [None req-ade7888a-18d7-44c7-9f8d-23d1865c16ae tempest-FloatingIPsAssociationNegativeTestJSON-1245561580 tempest-FloatingIPsAssociationNegativeTestJSON-1245561580-project-member] [instance: ea496eae-68f9-43f1-b4cf-6743043c753b] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 553.958304] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Expecting reply to msg e97d67d8d37246fa875d29eb96b646e9 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 554.120406] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg e97d67d8d37246fa875d29eb96b646e9 [ 554.460779] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 554.475901] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 554.476278] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 554.476624] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 554.564895] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 554.565127] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 554.565293] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 554.565663] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 554.565839] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 554.565994] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 554.580013] nova-conductor[53689]: DEBUG nova.quota [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Getting quotas for project 9a626302e1fb47f282f9cff6b397713c. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 554.582162] nova-conductor[53689]: DEBUG nova.quota [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Getting quotas for user 61307ba65ba9474ebf19305ed6fb735d and project 9a626302e1fb47f282f9cff6b397713c. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 554.587366] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] [instance: 782dc85a-56f4-4f03-8711-b78bbadb33ce] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 554.587951] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 554.588252] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 554.588423] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 554.593623] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] [instance: 782dc85a-56f4-4f03-8711-b78bbadb33ce] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 554.594346] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 554.594543] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 554.594704] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 554.617413] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 554.617641] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 554.617803] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 554.779322] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-ccd4ba1d-9601-49d3-9327-4721d6411aba tempest-ServersAdmin275Test-1388887324 tempest-ServersAdmin275Test-1388887324-project-admin] Targeting cell 84364d98-e6ce-480f-8cca-682929b0326f(cell1) for conductor method rebuild_instance {{(pid=53689) wrapper /opt/stack/nova/nova/conductor/manager.py:95}} [ 554.779587] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ccd4ba1d-9601-49d3-9327-4721d6411aba tempest-ServersAdmin275Test-1388887324 tempest-ServersAdmin275Test-1388887324-project-admin] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 554.779790] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ccd4ba1d-9601-49d3-9327-4721d6411aba tempest-ServersAdmin275Test-1388887324 tempest-ServersAdmin275Test-1388887324-project-admin] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 554.779975] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ccd4ba1d-9601-49d3-9327-4721d6411aba tempest-ServersAdmin275Test-1388887324 tempest-ServersAdmin275Test-1388887324-project-admin] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 554.825638] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-ccd4ba1d-9601-49d3-9327-4721d6411aba tempest-ServersAdmin275Test-1388887324 tempest-ServersAdmin275Test-1388887324-project-admin] [instance: 4d75a1ec-6c11-4cdd-ba0e-aa6d3add80f2] No migration record for the rebuild/evacuate request. {{(pid=53689) rebuild_instance /opt/stack/nova/nova/conductor/manager.py:1227}} [ 558.360503] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] [instance: f58fbc9b-ba60-4c56-bbdf-fbaf45ec12b5] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port caf04a4d-9a60-4997-9d4d-89dfc0470281, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance f58fbc9b-ba60-4c56-bbdf-fbaf45ec12b5 was re-scheduled: Binding failed for port caf04a4d-9a60-4997-9d4d-89dfc0470281, please check neutron logs for more information.\n'] [ 558.361723] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 558.361982] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f58fbc9b-ba60-4c56-bbdf-fbaf45ec12b5.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f58fbc9b-ba60-4c56-bbdf-fbaf45ec12b5. [ 558.363602] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] [instance: f58fbc9b-ba60-4c56-bbdf-fbaf45ec12b5] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f58fbc9b-ba60-4c56-bbdf-fbaf45ec12b5. [ 558.395960] nova-conductor[53688]: DEBUG nova.network.neutron [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] [instance: f58fbc9b-ba60-4c56-bbdf-fbaf45ec12b5] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 558.411953] nova-conductor[53688]: DEBUG nova.network.neutron [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] [instance: f58fbc9b-ba60-4c56-bbdf-fbaf45ec12b5] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 558.416132] nova-conductor[53688]: DEBUG nova.network.neutron [None req-49624888-9b63-46d8-9568-cc2a60de73dc tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] [instance: f58fbc9b-ba60-4c56-bbdf-fbaf45ec12b5] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 558.542912] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Expecting reply to msg 945d5e4d9ac243eeb47cfe7591388f6a in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 558.772585] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 945d5e4d9ac243eeb47cfe7591388f6a [ 559.048041] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 559.073816] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 559.073816] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 559.073816] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 559.113288] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 559.113516] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 559.113679] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 559.114043] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 559.114216] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 559.114367] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 559.126198] nova-conductor[53689]: DEBUG nova.quota [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Getting quotas for project 8e691198c7b14436bfc38c8b97b9519d. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 559.128965] nova-conductor[53689]: DEBUG nova.quota [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Getting quotas for user 906e8368f31e4815892c2b75e91f9d27 and project 8e691198c7b14436bfc38c8b97b9519d. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 559.135218] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] [instance: 1336becb-9691-490c-86ea-3bc70d13d7df] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 559.135698] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 559.135900] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 559.138815] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 559.138981] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] [instance: 1336becb-9691-490c-86ea-3bc70d13d7df] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 559.139905] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 559.140116] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 559.141293] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 559.162740] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 559.162992] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 559.163160] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 559.807451] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] [instance: 40709d2b-cc33-4ac0-9a13-731442d7edff] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 916f0a29-ac67-4ea8-b7f9-7d34e8b0b737, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 40709d2b-cc33-4ac0-9a13-731442d7edff was re-scheduled: Binding failed for port 916f0a29-ac67-4ea8-b7f9-7d34e8b0b737, please check neutron logs for more information.\n'] [ 559.808906] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 559.808906] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 40709d2b-cc33-4ac0-9a13-731442d7edff.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 40709d2b-cc33-4ac0-9a13-731442d7edff. [ 559.809078] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] [instance: 40709d2b-cc33-4ac0-9a13-731442d7edff] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 40709d2b-cc33-4ac0-9a13-731442d7edff. [ 559.846808] nova-conductor[53688]: DEBUG nova.network.neutron [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] [instance: 40709d2b-cc33-4ac0-9a13-731442d7edff] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 559.881233] nova-conductor[53688]: DEBUG nova.network.neutron [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] [instance: 40709d2b-cc33-4ac0-9a13-731442d7edff] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 559.891443] nova-conductor[53688]: DEBUG nova.network.neutron [None req-d7699baf-a8e6-40a5-be5b-f44995055732 tempest-ServersAdminTestJSON-923997297 tempest-ServersAdminTestJSON-923997297-project-member] [instance: 40709d2b-cc33-4ac0-9a13-731442d7edff] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 563.070941] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Expecting reply to msg 5b1328c53fa24ac18355a284ef7ff46d in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 563.222834] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 5b1328c53fa24ac18355a284ef7ff46d [ 563.574169] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 563.592519] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 563.593157] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 563.593157] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 563.646231] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 563.646495] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 563.646673] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 563.647061] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 563.647256] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 563.647417] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 563.660210] nova-conductor[53688]: DEBUG nova.quota [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Getting quotas for project c95c48da3e454ce4b42d95ae381f16ed. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 563.685914] nova-conductor[53688]: DEBUG nova.quota [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Getting quotas for user 8742f4ce8dd94720a20be2ae3f9e4e1c and project c95c48da3e454ce4b42d95ae381f16ed. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 563.699856] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] [instance: 30ed4438-4f74-4bc3-a6cc-a59420751940] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 563.700535] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 563.700909] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 563.701220] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 563.743979] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] [instance: 30ed4438-4f74-4bc3-a6cc-a59420751940] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 563.744754] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 563.744972] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 563.745197] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 563.802904] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 563.803046] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 563.803309] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 565.466024] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Expecting reply to msg daa81acb4bd9416eaa9148b2bfef1a42 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 565.631071] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg daa81acb4bd9416eaa9148b2bfef1a42 [ 565.969623] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 565.985032] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 565.985374] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 565.985464] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 566.030651] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] Expecting reply to msg 78c79e90cf6847b98f74b2b743a6f07b in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 566.047581] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 566.047888] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 566.048133] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 566.048534] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 566.048750] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 566.048941] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 566.062290] nova-conductor[53688]: DEBUG nova.quota [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Getting quotas for project fc437f0153524c71b3212518dc00bd2e. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 566.066194] nova-conductor[53688]: DEBUG nova.quota [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Getting quotas for user e298aba9bd1946fabb634ea5c111c634 and project fc437f0153524c71b3212518dc00bd2e. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 566.075160] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] [instance: 767d7956-954b-4be7-8cc6-45872ff4cfce] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 566.075699] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 566.075953] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 566.076213] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 566.083738] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] [instance: 767d7956-954b-4be7-8cc6-45872ff4cfce] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 566.084401] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 566.084647] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 566.084815] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 566.111103] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 566.111379] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 566.111651] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 566.193010] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 78c79e90cf6847b98f74b2b743a6f07b [ 566.535601] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 566.551832] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 566.552052] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 566.552355] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 566.621612] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 566.621854] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 566.622023] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 566.622385] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 566.622574] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 566.622812] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 566.642128] nova-conductor[53688]: DEBUG nova.quota [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] Getting quotas for project b8db06aa9af6414d814c84d2bb435052. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 566.645875] nova-conductor[53688]: DEBUG nova.quota [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] Getting quotas for user 06261b7b845d45998532157d562152ae and project b8db06aa9af6414d814c84d2bb435052. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 566.652733] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] [instance: 109fc11e-d640-4617-99a3-0defe0a5aa6c] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 566.653426] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 566.653592] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 566.653716] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 566.661412] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] [instance: 109fc11e-d640-4617-99a3-0defe0a5aa6c] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 566.662804] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 566.663296] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 566.663497] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 566.715279] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 566.715794] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 566.716207] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-20b84c65-f886-4d64-81dc-131def0d4b19 tempest-ServersAaction247Test-1023329623 tempest-ServersAaction247Test-1023329623-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 567.875080] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Expecting reply to msg 254ada10571f4d93a678217bece40b1e in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 568.038370] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 254ada10571f4d93a678217bece40b1e [ 568.378455] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 568.401984] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 568.402232] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 568.402800] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 568.442496] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 568.442733] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 568.442929] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 568.443306] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 568.443531] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 568.443638] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 568.457990] nova-conductor[53689]: DEBUG nova.quota [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Getting quotas for project c7f053eccf8842afb90d2bece96f9311. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 568.461246] nova-conductor[53689]: DEBUG nova.quota [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Getting quotas for user ceb5d8ab82344fe9baf8b970bf3a475a and project c7f053eccf8842afb90d2bece96f9311. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 568.472347] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] [instance: b41c6d21-5e7f-427f-95ce-830fe0da8bc6] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 568.472969] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 568.473187] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 568.473378] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 568.477769] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] [instance: b41c6d21-5e7f-427f-95ce-830fe0da8bc6] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 568.478458] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 568.478674] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 568.478861] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 568.494924] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 568.495139] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 568.495308] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 570.165950] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SELECT=381,nova_cell1:UPDATE=163,nova_cell1:INSERT=28,nova_cell1:SAVEPOINT=28,nova_cell1:RELEASE=28 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 570.551449] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=338,nova_cell1:UPDATE=146,nova_cell1:INSERT=20,nova_cell1:SAVEPOINT=24,nova_cell1:RELEASE=24 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 573.123041] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=90,nova_api:UPDATE=5,nova_api:DELETE=7 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 573.131676] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Expecting reply to msg 8f2edf744f8c44cbb8a55bb3594632d5 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 573.351624] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 8f2edf744f8c44cbb8a55bb3594632d5 [ 573.634289] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 573.648906] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 573.649146] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 573.651147] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 573.713286] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 573.713522] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 573.713694] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 573.714073] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 573.714249] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 573.714404] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 573.733482] nova-conductor[53688]: DEBUG nova.quota [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Getting quotas for project 094287f1b5004fa8a13d982851ec9347. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 573.740403] nova-conductor[53688]: DEBUG nova.quota [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Getting quotas for user eaba9dda37534605bc42ae92453f87c6 and project 094287f1b5004fa8a13d982851ec9347. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 573.752633] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] [instance: 12c47e99-faf4-4083-a46f-4e33c451e980] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 573.753234] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 573.753447] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 573.753606] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 573.764544] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] [instance: 12c47e99-faf4-4083-a46f-4e33c451e980] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 573.765305] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 573.765514] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 573.765675] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 573.804778] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 573.805140] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 573.805690] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 574.947714] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Expecting reply to msg 2e6647ac58cb46399ac61724d1c1eba2 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 575.067835] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] [instance: 1784917b-8a7e-4974-b8b3-f8f2b3db019a] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 14197306-66ad-4077-914c-b79fd0a658db, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 1784917b-8a7e-4974-b8b3-f8f2b3db019a was re-scheduled: Binding failed for port 14197306-66ad-4077-914c-b79fd0a658db, please check neutron logs for more information.\n'] [ 575.067835] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 575.067835] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1784917b-8a7e-4974-b8b3-f8f2b3db019a.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1784917b-8a7e-4974-b8b3-f8f2b3db019a. [ 575.067835] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] [instance: 1784917b-8a7e-4974-b8b3-f8f2b3db019a] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1784917b-8a7e-4974-b8b3-f8f2b3db019a. [ 575.130242] nova-conductor[53689]: DEBUG nova.network.neutron [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] [instance: 1784917b-8a7e-4974-b8b3-f8f2b3db019a] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 575.180481] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 2e6647ac58cb46399ac61724d1c1eba2 [ 575.189483] nova-conductor[53689]: DEBUG nova.network.neutron [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] [instance: 1784917b-8a7e-4974-b8b3-f8f2b3db019a] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 575.200811] nova-conductor[53689]: DEBUG nova.network.neutron [None req-3eedc122-e763-4f29-b667-fc450f60f44b tempest-ServerRescueTestJSON-1950758181 tempest-ServerRescueTestJSON-1950758181-project-member] [instance: 1784917b-8a7e-4974-b8b3-f8f2b3db019a] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 575.452783] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 575.466241] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 575.466635] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 575.466635] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 575.532432] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 575.532684] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 575.532855] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 575.533218] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 575.533399] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 575.533555] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 575.545983] nova-conductor[53688]: DEBUG nova.quota [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Getting quotas for project f484798dc24b465ba441d235e2943348. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 575.549151] nova-conductor[53688]: DEBUG nova.quota [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Getting quotas for user 824e8c3c583b4bf39d5d386d7fe06ea6 and project f484798dc24b465ba441d235e2943348. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 575.555019] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] [instance: 5ea287cd-ba85-446d-85d0-5a050fe49f17] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 575.555019] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 575.555019] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 575.555272] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 575.559832] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] [instance: 5ea287cd-ba85-446d-85d0-5a050fe49f17] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 575.560573] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 575.560777] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 575.560880] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 575.594377] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 575.594603] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 575.594808] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 577.118597] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Expecting reply to msg ad97981a695b498b85e2e2372bbfd26f in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 577.270063] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg ad97981a695b498b85e2e2372bbfd26f [ 577.620821] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 577.622986] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=87,nova_api:UPDATE=10,nova_api:DELETE=7 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 577.636563] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 577.636790] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 577.637080] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 577.639743] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SELECT=50,nova_cell1:SAVEPOINT=14,nova_cell1:INSERT=128,nova_cell1:RELEASE=14 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 577.728127] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 577.728127] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 577.728127] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 577.728127] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 577.728127] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 577.728127] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 577.731924] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell0:SELECT=18 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 577.748448] nova-conductor[53688]: DEBUG nova.quota [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Getting quotas for project d9e32c9f8505420595cdd8f18285a33f. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 577.752478] nova-conductor[53688]: DEBUG nova.quota [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Getting quotas for user 6c24c9f3c3594981b6c9970cd8c8f877 and project d9e32c9f8505420595cdd8f18285a33f. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 577.759495] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] [instance: 297a5546-6159-462c-a436-032d94855c00] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 577.760078] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 577.760260] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 577.760429] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 577.784416] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] [instance: 297a5546-6159-462c-a436-032d94855c00] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 577.785161] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 577.785383] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 577.785728] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 577.826834] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 577.827073] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 577.827239] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 578.456136] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell0:SELECT=18 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 578.481271] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=61,nova_cell1:SAVEPOINT=15,nova_cell1:INSERT=137,nova_cell1:RELEASE=15,nova_cell1:UPDATE=6 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 578.485647] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=106,nova_api:UPDATE=13,nova_api:DELETE=8 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 578.501612] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=101,nova_api:UPDATE=4,nova_api:DELETE=9 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 579.500407] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] [instance: 228821ca-e981-405b-8952-8a1718103d3c] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port f8f7383d-3e1b-4747-82b6-44e40294ada9, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 228821ca-e981-405b-8952-8a1718103d3c was re-scheduled: Binding failed for port f8f7383d-3e1b-4747-82b6-44e40294ada9, please check neutron logs for more information.\n'] [ 579.500407] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 579.500885] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 228821ca-e981-405b-8952-8a1718103d3c.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 228821ca-e981-405b-8952-8a1718103d3c. [ 579.500885] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] [instance: 228821ca-e981-405b-8952-8a1718103d3c] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 228821ca-e981-405b-8952-8a1718103d3c. [ 579.532368] nova-conductor[53689]: DEBUG nova.network.neutron [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] [instance: 228821ca-e981-405b-8952-8a1718103d3c] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 579.618852] nova-conductor[53689]: DEBUG nova.network.neutron [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] [instance: 228821ca-e981-405b-8952-8a1718103d3c] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 579.624889] nova-conductor[53689]: DEBUG nova.network.neutron [None req-3fde7998-9712-420b-a423-b457db635f29 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] [instance: 228821ca-e981-405b-8952-8a1718103d3c] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 583.478292] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Expecting reply to msg 9b0c44bb2c9e413487d6561176836b6f in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 583.669250] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 9b0c44bb2c9e413487d6561176836b6f [ 583.979957] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 584.007223] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 584.007923] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 584.008277] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 584.071674] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 584.071770] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 584.071887] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 584.072744] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 584.072941] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 584.073103] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 584.122081] nova-conductor[53688]: DEBUG nova.quota [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Getting quotas for project 67f289a0c54e46e18b97b47c11a9b1f5. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 584.126563] nova-conductor[53688]: DEBUG nova.quota [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Getting quotas for user 6116fae5f64e41e09dd7d6f6fcab88dd and project 67f289a0c54e46e18b97b47c11a9b1f5. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 584.133939] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] [instance: 2b87dfbe-2b94-4787-a795-94f8b63f651c] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 584.134370] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 584.134804] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 584.136976] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 584.143932] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] [instance: 2b87dfbe-2b94-4787-a795-94f8b63f651c] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 584.144645] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 584.144868] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 584.145042] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 584.162368] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 584.162555] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 584.162728] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 586.064936] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Expecting reply to msg d9b58721e65c40fea70d90b786da69f9 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 586.258314] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg d9b58721e65c40fea70d90b786da69f9 [ 586.572353] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 586.590835] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 586.591107] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 586.591300] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 586.625404] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 586.626028] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 586.626028] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 586.626167] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 586.626343] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 586.626502] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 586.634955] nova-conductor[53688]: DEBUG nova.quota [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Getting quotas for project f2a584ced06549eaa9449580059e9b1c. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 586.637232] nova-conductor[53688]: DEBUG nova.quota [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Getting quotas for user dca60306b0944a8ba1bfd91c00439602 and project f2a584ced06549eaa9449580059e9b1c. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 586.648112] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] [instance: e2560c8e-61c6-4343-82cb-47dc5b1997fb] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 586.648622] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 586.648823] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 586.648992] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 586.653941] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] [instance: e2560c8e-61c6-4343-82cb-47dc5b1997fb] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 586.654672] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 586.654874] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 586.661692] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 586.726654] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 586.726963] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 586.727148] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 595.855293] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 595.855597] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 595.855829] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 595.878456] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Expecting reply to msg 6086cd5691d54681901bb1ae33e25ad3 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 596.040508] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 6086cd5691d54681901bb1ae33e25ad3 [ 596.382337] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 596.395177] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 596.395401] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 596.395566] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 596.437474] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 596.437700] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 596.437872] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 596.438238] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 596.438422] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 596.438583] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 596.446270] nova-conductor[53689]: DEBUG nova.quota [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Getting quotas for project 683850b1cecd48ad9939566a9745c1aa. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 596.448361] nova-conductor[53689]: DEBUG nova.quota [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Getting quotas for user c29f650cbd91401d995341c03f795e88 and project 683850b1cecd48ad9939566a9745c1aa. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 596.453256] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] [instance: 3635532a-2af3-4ef5-a922-37fc763c9708] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 596.453683] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 596.453885] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 596.454044] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 596.458066] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] [instance: 3635532a-2af3-4ef5-a922-37fc763c9708] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 596.458703] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 596.458899] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 596.459062] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 596.480033] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 596.480033] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 596.480033] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 596.631517] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell0:SELECT=2 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 596.666437] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SAVEPOINT=2,nova_cell1:INSERT=23,nova_cell1:RELEASE=2,nova_cell1:SELECT=8 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 596.681953] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=31,nova_api:DELETE=3,nova_api:UPDATE=3 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 596.723805] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=19,nova_api:UPDATE=1,nova_api:DELETE=2 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 598.606514] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] [instance: 1d64d913-45f0-4768-8375-7863d9ae43c3] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port d803f7f9-4edd-478d-8efa-d1db78feae38, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 1d64d913-45f0-4768-8375-7863d9ae43c3 was re-scheduled: Binding failed for port d803f7f9-4edd-478d-8efa-d1db78feae38, please check neutron logs for more information.\n'] [ 598.607615] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 598.607882] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1d64d913-45f0-4768-8375-7863d9ae43c3.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1d64d913-45f0-4768-8375-7863d9ae43c3. [ 598.610578] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] [instance: 1d64d913-45f0-4768-8375-7863d9ae43c3] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1d64d913-45f0-4768-8375-7863d9ae43c3. [ 598.637062] nova-conductor[53689]: DEBUG nova.network.neutron [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] [instance: 1d64d913-45f0-4768-8375-7863d9ae43c3] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 598.680560] nova-conductor[53689]: DEBUG nova.network.neutron [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] [instance: 1d64d913-45f0-4768-8375-7863d9ae43c3] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 598.684659] nova-conductor[53689]: DEBUG nova.network.neutron [None req-ad704dec-d447-49f2-84f8-39ab723d8651 tempest-VolumesAssistedSnapshotsTest-325362177 tempest-VolumesAssistedSnapshotsTest-325362177-project-member] [instance: 1d64d913-45f0-4768-8375-7863d9ae43c3] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 601.196736] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Expecting reply to msg 6fb66399ac17435d80505641ba476973 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 601.355663] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 6fb66399ac17435d80505641ba476973 [ 601.702618] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 601.723105] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.723339] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.723514] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.765089] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.765357] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.765571] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.765956] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.766215] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.766400] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.777358] nova-conductor[53689]: DEBUG nova.quota [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Getting quotas for project 039a458fcf5642eaae5d4353f569d660. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 601.783011] nova-conductor[53689]: DEBUG nova.quota [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Getting quotas for user e785a722b76c42ac91c04e9e9958183f and project 039a458fcf5642eaae5d4353f569d660. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 601.791213] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] [instance: 144c3c21-b18e-4997-a241-8ff21a3b4835] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 601.791858] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.792102] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.792278] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.798024] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] [instance: 144c3c21-b18e-4997-a241-8ff21a3b4835] block_device_mapping [BlockDeviceMapping(attachment_id=8942fc23-3d28-4b18-a8c9-56af0c2915b3,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='volume',device_name=None,device_type=None,disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id=None,instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='volume',tag=None,updated_at=,uuid=,volume_id='770432a3-4ca1-4717-9890-c964bd2d2419',volume_size=1,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 601.798765] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.799222] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.799601] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 601.836138] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 601.836470] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 601.836714] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 604.430153] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Expecting reply to msg 588ab3cfd38e491f86111b78a6cbe6ca in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 604.572214] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 588ab3cfd38e491f86111b78a6cbe6ca [ 604.933039] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 604.949994] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 604.950251] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 604.950425] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 605.004880] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 605.005118] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 605.005291] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 605.005665] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 605.005845] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 605.006003] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 605.016896] nova-conductor[53688]: DEBUG nova.quota [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Getting quotas for project 233511ca341849e3a8d33ee8152f429a. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 605.020629] nova-conductor[53688]: DEBUG nova.quota [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Getting quotas for user b3174ce41afd40c188eb376297330d8c and project 233511ca341849e3a8d33ee8152f429a. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 605.026052] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] [instance: 8d63e0a8-85a1-400b-a6f0-8e87c7945655] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 605.026607] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 605.026850] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 605.027052] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 605.033655] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] [instance: 8d63e0a8-85a1-400b-a6f0-8e87c7945655] block_device_mapping [BlockDeviceMapping(attachment_id=baf3a147-1844-4f03-9b94-f8915669e405,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='volume',device_name=None,device_type=None,disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id=None,instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='volume',tag=None,updated_at=,uuid=,volume_id='4602f78a-d619-4e07-8958-2029cb61c217',volume_size=1,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 605.034436] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 605.034881] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 605.034881] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 605.052659] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 605.053093] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 605.053154] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 605.096404] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] [instance: f0c26eb3-e6d6-4d9f-9f07-5add9de6d126] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port bc737d73-0b13-4405-8b56-e8520c00a00f, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance f0c26eb3-e6d6-4d9f-9f07-5add9de6d126 was re-scheduled: Binding failed for port bc737d73-0b13-4405-8b56-e8520c00a00f, please check neutron logs for more information.\n'] [ 605.097178] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 605.097288] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f0c26eb3-e6d6-4d9f-9f07-5add9de6d126.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f0c26eb3-e6d6-4d9f-9f07-5add9de6d126. [ 605.097764] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] [instance: f0c26eb3-e6d6-4d9f-9f07-5add9de6d126] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f0c26eb3-e6d6-4d9f-9f07-5add9de6d126. [ 605.133552] nova-conductor[53689]: DEBUG nova.network.neutron [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] [instance: f0c26eb3-e6d6-4d9f-9f07-5add9de6d126] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 605.187722] nova-conductor[53689]: DEBUG nova.network.neutron [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] [instance: f0c26eb3-e6d6-4d9f-9f07-5add9de6d126] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 605.196298] nova-conductor[53689]: DEBUG nova.network.neutron [None req-6ab6dead-0874-4403-9491-2b16e11b1675 tempest-AttachInterfacesV270Test-2098825571 tempest-AttachInterfacesV270Test-2098825571-project-member] [instance: f0c26eb3-e6d6-4d9f-9f07-5add9de6d126] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 609.604538] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Expecting reply to msg aed756e55a004d848aca30b9ccf2ece1 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 609.792186] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg aed756e55a004d848aca30b9ccf2ece1 [ 610.108398] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 610.125614] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 610.126275] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 610.126449] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 610.170535] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 610.170766] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 610.170933] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 610.171296] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 610.171477] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 610.171664] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 610.181556] nova-conductor[53688]: DEBUG nova.quota [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Getting quotas for project c6ac2fa4041e4f24bee18db89143d7dc. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 610.185088] nova-conductor[53688]: DEBUG nova.quota [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Getting quotas for user 438a167fe61344bc9371e3b42d5344c8 and project c6ac2fa4041e4f24bee18db89143d7dc. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 610.190255] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] [instance: bf40cc8c-4729-49c5-8c9d-e3ee09606aa5] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 610.190754] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 610.190953] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 610.191116] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 610.198133] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] [instance: bf40cc8c-4729-49c5-8c9d-e3ee09606aa5] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 610.198892] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 610.199108] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 610.199280] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 610.221638] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 610.221965] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 610.222166] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 611.771370] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell0:SELECT=2 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 611.808917] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=6,nova_cell1:SAVEPOINT=2,nova_cell1:INSERT=16,nova_cell1:RELEASE=2 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 611.823014] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=15 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 611.832436] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=10,nova_api:UPDATE=2,nova_api:DELETE=2 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 612.019134] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] [instance: 8f37dfb3-c4d2-4c41-91eb-e3c7ca640050] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port a5e6f22e-f786-4eb3-9df7-8e5f76d3b84e, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 8f37dfb3-c4d2-4c41-91eb-e3c7ca640050 was re-scheduled: Binding failed for port a5e6f22e-f786-4eb3-9df7-8e5f76d3b84e, please check neutron logs for more information.\n'] [ 612.019819] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 612.020069] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 8f37dfb3-c4d2-4c41-91eb-e3c7ca640050.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 8f37dfb3-c4d2-4c41-91eb-e3c7ca640050. [ 612.020290] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] [instance: 8f37dfb3-c4d2-4c41-91eb-e3c7ca640050] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 8f37dfb3-c4d2-4c41-91eb-e3c7ca640050. [ 612.065550] nova-conductor[53689]: DEBUG nova.network.neutron [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] [instance: 8f37dfb3-c4d2-4c41-91eb-e3c7ca640050] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 612.096417] nova-conductor[53689]: DEBUG nova.network.neutron [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] [instance: 8f37dfb3-c4d2-4c41-91eb-e3c7ca640050] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 612.101409] nova-conductor[53689]: DEBUG nova.network.neutron [None req-7e015aab-3e7e-485b-abca-dc5626d0be89 tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] [instance: 8f37dfb3-c4d2-4c41-91eb-e3c7ca640050] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 613.604598] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Expecting reply to msg cc463b8d634743cfbc269286c146c445 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 613.791360] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg cc463b8d634743cfbc269286c146c445 [ 614.085446] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] [instance: 3f1f549f-8034-4685-b6f0-db5a7a2a4a32] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 2c9b8cd4-1c30-438e-a736-fd3acb6e41ac, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 3f1f549f-8034-4685-b6f0-db5a7a2a4a32 was re-scheduled: Binding failed for port 2c9b8cd4-1c30-438e-a736-fd3acb6e41ac, please check neutron logs for more information.\n'] [ 614.086058] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 614.086290] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 3f1f549f-8034-4685-b6f0-db5a7a2a4a32.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 3f1f549f-8034-4685-b6f0-db5a7a2a4a32. [ 614.086609] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] [instance: 3f1f549f-8034-4685-b6f0-db5a7a2a4a32] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 3f1f549f-8034-4685-b6f0-db5a7a2a4a32. [ 614.106984] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 614.109199] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Expecting reply to msg 38d3085af8ad44b98b4cb07599e69025 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 614.136590] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 614.136590] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 614.136590] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 614.142257] nova-conductor[53688]: DEBUG nova.network.neutron [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] [instance: 3f1f549f-8034-4685-b6f0-db5a7a2a4a32] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 614.168440] nova-conductor[53688]: DEBUG nova.network.neutron [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] [instance: 3f1f549f-8034-4685-b6f0-db5a7a2a4a32] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 614.184415] nova-conductor[53688]: DEBUG nova.network.neutron [None req-ff4d2de5-6719-470e-b0fd-ff1700045390 tempest-ServerActionsTestOtherA-1252012706 tempest-ServerActionsTestOtherA-1252012706-project-member] [instance: 3f1f549f-8034-4685-b6f0-db5a7a2a4a32] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 614.200951] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 614.201188] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 614.201490] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 614.202021] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 614.202021] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 614.202129] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 614.209773] nova-conductor[53689]: DEBUG nova.quota [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Getting quotas for project d6c92f944a2b4fb88208d093a86b5a93. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 614.212777] nova-conductor[53689]: DEBUG nova.quota [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Getting quotas for user cc86d52877f44ed9818e3455d93bacfc and project d6c92f944a2b4fb88208d093a86b5a93. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 614.217787] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] [instance: 23774aa5-1608-495f-8015-29e25f856c69] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 614.218220] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 614.218456] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 614.218586] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 614.223667] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] [instance: 23774aa5-1608-495f-8015-29e25f856c69] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 614.224377] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 614.224656] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 614.224826] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 614.241839] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 614.242079] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 614.242242] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 614.274231] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 38d3085af8ad44b98b4cb07599e69025 [ 614.611807] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 614.625943] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 614.626172] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 614.626345] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 614.666052] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 614.666294] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 614.666462] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 614.666977] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 614.667161] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 614.667320] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 614.693201] nova-conductor[53688]: DEBUG nova.quota [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Getting quotas for project 03a137cec9c540259688f75fb14dce35. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 614.701465] nova-conductor[53688]: DEBUG nova.quota [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Getting quotas for user 9e14d098d2604a1b9543150852012a1a and project 03a137cec9c540259688f75fb14dce35. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 614.719495] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] [instance: b6a158f8-6e2a-4967-ad05-761804ec6590] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 614.720270] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 614.720857] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 614.721093] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 614.736115] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] [instance: b6a158f8-6e2a-4967-ad05-761804ec6590] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 614.736913] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 614.744321] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 614.744551] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.007s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 614.878880] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 614.879136] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 614.879311] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 615.990679] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] [instance: 36c3ac75-5bfd-4a89-9ddb-28fded8da39c] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port a2439eab-fa0c-4c49-9830-697e9a092f05, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 36c3ac75-5bfd-4a89-9ddb-28fded8da39c was re-scheduled: Binding failed for port a2439eab-fa0c-4c49-9830-697e9a092f05, please check neutron logs for more information.\n'] [ 615.991292] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 615.991516] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 36c3ac75-5bfd-4a89-9ddb-28fded8da39c.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 36c3ac75-5bfd-4a89-9ddb-28fded8da39c. [ 615.991888] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] [instance: 36c3ac75-5bfd-4a89-9ddb-28fded8da39c] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 36c3ac75-5bfd-4a89-9ddb-28fded8da39c. [ 616.023535] nova-conductor[53689]: DEBUG nova.network.neutron [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] [instance: 36c3ac75-5bfd-4a89-9ddb-28fded8da39c] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 616.064685] nova-conductor[53689]: DEBUG nova.network.neutron [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] [instance: 36c3ac75-5bfd-4a89-9ddb-28fded8da39c] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 616.067995] nova-conductor[53689]: DEBUG nova.network.neutron [None req-2b100f97-892f-472e-a7d9-9ca8350c8021 tempest-ServersWithSpecificFlavorTestJSON-970394519 tempest-ServersWithSpecificFlavorTestJSON-970394519-project-member] [instance: 36c3ac75-5bfd-4a89-9ddb-28fded8da39c] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 617.718029] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Expecting reply to msg ac58bb1596a24fe4aa3a10b5560615d0 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 617.883029] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg ac58bb1596a24fe4aa3a10b5560615d0 [ 618.221387] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 618.234839] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 618.235071] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 618.235243] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 618.277608] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 618.277916] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 618.278134] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 618.278561] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 618.278798] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 618.279277] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 618.291521] nova-conductor[53689]: DEBUG nova.quota [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Getting quotas for project 2160ae8026c4462bb19f174f578e8768. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 618.294622] nova-conductor[53689]: DEBUG nova.quota [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Getting quotas for user 2ae8205c40084bb78bfbb5ae2291781c and project 2160ae8026c4462bb19f174f578e8768. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 618.300763] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] [instance: ca8a38c7-a81c-407a-9558-3d15e492d9fa] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 618.301248] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 618.301601] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 618.301830] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 618.306249] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] [instance: ca8a38c7-a81c-407a-9558-3d15e492d9fa] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 618.309845] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 618.309845] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 618.309845] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 618.323519] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] [instance: 5cddeea1-7558-4c12-afdc-2ea7a706881a] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 319ef93a-07c4-43fa-885a-2600407961b4, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 5cddeea1-7558-4c12-afdc-2ea7a706881a was re-scheduled: Binding failed for port 319ef93a-07c4-43fa-885a-2600407961b4, please check neutron logs for more information.\n'] [ 618.324585] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 618.325009] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 5cddeea1-7558-4c12-afdc-2ea7a706881a.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 5cddeea1-7558-4c12-afdc-2ea7a706881a. [ 618.325438] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] [instance: 5cddeea1-7558-4c12-afdc-2ea7a706881a] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 5cddeea1-7558-4c12-afdc-2ea7a706881a. [ 618.347524] nova-conductor[53689]: DEBUG nova.network.neutron [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] [instance: 5cddeea1-7558-4c12-afdc-2ea7a706881a] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 618.351351] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 618.351517] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 618.351723] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-99d429a5-56ce-4340-aadf-504939f66379 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 618.365106] nova-conductor[53689]: DEBUG nova.network.neutron [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] [instance: 5cddeea1-7558-4c12-afdc-2ea7a706881a] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 618.369540] nova-conductor[53689]: DEBUG nova.network.neutron [None req-ae7e47f1-e831-45dc-b4b0-3acce34fffb3 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] [instance: 5cddeea1-7558-4c12-afdc-2ea7a706881a] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 618.432957] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Expecting reply to msg fe1d136589244db2ad331ff81bee6260 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 618.553987] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg fe1d136589244db2ad331ff81bee6260 [ 618.935317] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 618.951072] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 618.951376] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 618.951610] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 618.995917] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 618.996306] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 618.996509] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 618.996941] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 618.997179] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 618.997392] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 619.010527] nova-conductor[53689]: DEBUG nova.quota [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Getting quotas for project 2160ae8026c4462bb19f174f578e8768. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 619.015116] nova-conductor[53689]: DEBUG nova.quota [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Getting quotas for user 2ae8205c40084bb78bfbb5ae2291781c and project 2160ae8026c4462bb19f174f578e8768. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 619.020447] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] [instance: 1efd9cfe-3a0c-412c-aa44-3bf650d08f9d] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 619.021049] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 619.021348] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 619.021701] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 619.024526] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] [instance: 1efd9cfe-3a0c-412c-aa44-3bf650d08f9d] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 619.025222] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 619.025475] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 619.025693] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 619.040626] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 619.040933] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 619.041164] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-89f69888-e995-47ed-8ffb-76173eb68b00 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 619.681306] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Expecting reply to msg 2f5b3d4e42a34bd99b45165e565720d2 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 619.834948] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 2f5b3d4e42a34bd99b45165e565720d2 [ 620.185830] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 620.198332] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 620.198571] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 620.198735] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 620.228788] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 620.228788] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 620.228788] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 620.228788] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 620.228788] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 620.228788] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 620.239358] nova-conductor[53688]: DEBUG nova.quota [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Getting quotas for project ca33c1550e694c38a15f9be28d24b19c. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 620.240294] nova-conductor[53688]: DEBUG nova.quota [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Getting quotas for user 0986847129324981ba5df690cc76bcba and project ca33c1550e694c38a15f9be28d24b19c. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 620.245900] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] [instance: 799d4a06-f7a3-4b92-8e96-ac076848fdd3] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 620.246371] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 620.246569] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 620.246731] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 620.251087] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] [instance: 799d4a06-f7a3-4b92-8e96-ac076848fdd3] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 620.251748] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 620.251951] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 620.252138] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 620.272260] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 620.272260] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 620.272557] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 620.474781] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] [instance: 7f8b08d4-3535-48ab-ba3f-a159511e2a64] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 6c735bdb-9615-424c-b35d-b618b55a0ca8, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 7f8b08d4-3535-48ab-ba3f-a159511e2a64 was re-scheduled: Binding failed for port 6c735bdb-9615-424c-b35d-b618b55a0ca8, please check neutron logs for more information.\n'] [ 620.475674] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 620.476474] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 7f8b08d4-3535-48ab-ba3f-a159511e2a64.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 7f8b08d4-3535-48ab-ba3f-a159511e2a64. [ 620.477048] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] [instance: 7f8b08d4-3535-48ab-ba3f-a159511e2a64] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 7f8b08d4-3535-48ab-ba3f-a159511e2a64. [ 620.517036] nova-conductor[53689]: DEBUG nova.network.neutron [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] [instance: 7f8b08d4-3535-48ab-ba3f-a159511e2a64] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 620.551718] nova-conductor[53689]: DEBUG nova.network.neutron [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] [instance: 7f8b08d4-3535-48ab-ba3f-a159511e2a64] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 620.556577] nova-conductor[53689]: DEBUG nova.network.neutron [None req-57518784-2066-4fc9-af51-753b31e55ef6 tempest-ServerRescueNegativeTestJSON-1657136755 tempest-ServerRescueNegativeTestJSON-1657136755-project-member] [instance: 7f8b08d4-3535-48ab-ba3f-a159511e2a64] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 622.420599] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] [instance: 7d1f7566-8e5e-476c-9d19-49ed7b16c308] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 8061659c-a94d-4f7d-a527-0f760ebc0807, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 7d1f7566-8e5e-476c-9d19-49ed7b16c308 was re-scheduled: Binding failed for port 8061659c-a94d-4f7d-a527-0f760ebc0807, please check neutron logs for more information.\n'] [ 622.420599] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 622.421006] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 7d1f7566-8e5e-476c-9d19-49ed7b16c308.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 7d1f7566-8e5e-476c-9d19-49ed7b16c308. [ 622.421006] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] [instance: 7d1f7566-8e5e-476c-9d19-49ed7b16c308] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 7d1f7566-8e5e-476c-9d19-49ed7b16c308. [ 622.445926] nova-conductor[53689]: DEBUG nova.network.neutron [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] [instance: 7d1f7566-8e5e-476c-9d19-49ed7b16c308] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 622.507420] nova-conductor[53689]: DEBUG nova.network.neutron [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] [instance: 7d1f7566-8e5e-476c-9d19-49ed7b16c308] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 622.511196] nova-conductor[53689]: DEBUG nova.network.neutron [None req-59065584-7b1c-496c-a06f-36029ba615b1 tempest-ServerDiagnosticsTest-1786720603 tempest-ServerDiagnosticsTest-1786720603-project-member] [instance: 7d1f7566-8e5e-476c-9d19-49ed7b16c308] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 627.072615] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: 1d84bcbd-9b55-4d6f-b6c7-24391c8600a7] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port eca6b818-5b64-4a66-a817-2488dc457863, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 1d84bcbd-9b55-4d6f-b6c7-24391c8600a7 was re-scheduled: Binding failed for port eca6b818-5b64-4a66-a817-2488dc457863, please check neutron logs for more information.\n'] [ 627.072615] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 627.073185] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1d84bcbd-9b55-4d6f-b6c7-24391c8600a7.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1d84bcbd-9b55-4d6f-b6c7-24391c8600a7. [ 627.073185] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: 1d84bcbd-9b55-4d6f-b6c7-24391c8600a7] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1d84bcbd-9b55-4d6f-b6c7-24391c8600a7. [ 627.104313] nova-conductor[53688]: DEBUG nova.network.neutron [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: 1d84bcbd-9b55-4d6f-b6c7-24391c8600a7] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 627.134008] nova-conductor[53688]: DEBUG nova.network.neutron [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: 1d84bcbd-9b55-4d6f-b6c7-24391c8600a7] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 627.141808] nova-conductor[53688]: DEBUG nova.network.neutron [None req-15bbf0be-5de5-4231-b593-9e16a99cd222 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: 1d84bcbd-9b55-4d6f-b6c7-24391c8600a7] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 628.551068] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Expecting reply to msg 66aabaa5b47043688cd3442a225d5e7e in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 628.697902] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 66aabaa5b47043688cd3442a225d5e7e [ 628.879511] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Expecting reply to msg e895bee6a394444abac05ccf21823ea4 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 629.002465] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell0:SELECT=3 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 629.029496] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=9,nova_cell1:SAVEPOINT=2,nova_cell1:INSERT=23,nova_cell1:RELEASE=2 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 629.038421] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=15,nova_api:UPDATE=2,nova_api:DELETE=2 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 629.055817] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 629.074143] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 629.074385] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 629.074566] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 629.098560] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg e895bee6a394444abac05ccf21823ea4 [ 629.108488] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 629.108719] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 629.108893] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 629.109381] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 629.109458] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 629.109597] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 629.117208] nova-conductor[53688]: DEBUG nova.quota [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Getting quotas for project 80cf26a75a1a420a85bdf5f723c98229. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 629.119798] nova-conductor[53688]: DEBUG nova.quota [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Getting quotas for user eee3c5a564a24daf92c4affc3607cbc7 and project 80cf26a75a1a420a85bdf5f723c98229. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 629.130672] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] [instance: 21213cff-55b3-48fd-91b4-6718f7819bc3] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 629.131118] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 629.131322] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 629.131480] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 629.140038] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] [instance: 21213cff-55b3-48fd-91b4-6718f7819bc3] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 629.140038] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 629.140038] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 629.140038] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 629.160112] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 629.160584] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 629.164651] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 629.387416] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 629.397784] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 629.398017] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 629.398183] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 629.418462] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 629.418682] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 629.418845] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 629.419198] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 629.419378] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 629.419549] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 629.427027] nova-conductor[53689]: DEBUG nova.quota [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Getting quotas for project a3de421e0f994df8b809ce0096753f23. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 629.429056] nova-conductor[53689]: DEBUG nova.quota [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Getting quotas for user c471664da5894985bf7478057ea19b73 and project a3de421e0f994df8b809ce0096753f23. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 629.434004] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: f3df4816-ef02-4ecc-a8ca-4f0eaf286218] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 629.434404] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 629.434599] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 629.434764] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 629.439874] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: f3df4816-ef02-4ecc-a8ca-4f0eaf286218] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 629.440496] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 629.440688] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 629.440851] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 629.457254] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 629.457540] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 629.457723] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 630.230235] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SAVEPOINT=35,nova_cell1:RELEASE=35,nova_cell1:UPDATE=158,nova_cell1:SELECT=348,nova_cell1:INSERT=16 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 630.631915] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=340,nova_cell1:UPDATE=188,nova_cell1:SAVEPOINT=35,nova_cell1:RELEASE=35,nova_cell1:INSERT=30,nova_cell1:DELETE=4 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 631.664629] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] [instance: 1804f229-97b9-4ee3-933d-715431a900f8] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 2a417537-62ce-4faf-b880-d5f49a9deb7f, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 1804f229-97b9-4ee3-933d-715431a900f8 was re-scheduled: Binding failed for port 2a417537-62ce-4faf-b880-d5f49a9deb7f, please check neutron logs for more information.\n'] [ 631.665213] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 631.665445] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1804f229-97b9-4ee3-933d-715431a900f8.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1804f229-97b9-4ee3-933d-715431a900f8. [ 631.665774] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] [instance: 1804f229-97b9-4ee3-933d-715431a900f8] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1804f229-97b9-4ee3-933d-715431a900f8. [ 631.707198] nova-conductor[53689]: DEBUG nova.network.neutron [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] [instance: 1804f229-97b9-4ee3-933d-715431a900f8] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 631.749411] nova-conductor[53689]: DEBUG nova.network.neutron [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] [instance: 1804f229-97b9-4ee3-933d-715431a900f8] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 631.755021] nova-conductor[53689]: DEBUG nova.network.neutron [None req-c40970f8-1d21-4690-85fe-1ed8a0091295 tempest-ServerMetadataNegativeTestJSON-932953273 tempest-ServerMetadataNegativeTestJSON-932953273-project-member] [instance: 1804f229-97b9-4ee3-933d-715431a900f8] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 633.214119] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Expecting reply to msg acc88c786a854e86b661b48ca8ab3d17 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 633.427758] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg acc88c786a854e86b661b48ca8ab3d17 [ 633.716690] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 633.732858] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 633.733181] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 633.733360] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 633.781051] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 633.781602] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 633.781602] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 633.781819] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 633.782048] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 633.782145] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 633.792793] nova-conductor[53689]: DEBUG nova.quota [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Getting quotas for project b2a065da05414c20bd187cbd0cd02b51. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 633.796065] nova-conductor[53689]: DEBUG nova.quota [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Getting quotas for user f6529f0831bb4b528385935bd06e9e8f and project b2a065da05414c20bd187cbd0cd02b51. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 633.800976] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] [instance: 2a7d4872-4ed7-4058-bc36-b199d89a9f14] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 633.801409] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 633.801628] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 633.801805] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 633.813988] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] [instance: 2a7d4872-4ed7-4058-bc36-b199d89a9f14] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 633.814678] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 633.814878] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 633.815043] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 633.863191] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 633.863415] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 633.863578] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 639.113033] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell0:SELECT=5 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 639.139026] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SELECT=15,nova_cell1:SAVEPOINT=4,nova_cell1:INSERT=40,nova_cell1:RELEASE=4 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 639.153307] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=34,nova_api:UPDATE=2,nova_api:DELETE=2 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 639.157219] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=26,nova_api:UPDATE=3,nova_api:DELETE=3 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 643.786204] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell0:SELECT=2 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 643.817084] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=6,nova_cell1:INSERT=15,nova_cell1:SAVEPOINT=1,nova_cell1:RELEASE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 643.856072] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=13,nova_api:DELETE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 643.859030] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=32,nova_api:UPDATE=3,nova_api:DELETE=2 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 648.508929] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 017b1da4-7c9b-477d-92a3-29b2248317d3] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port f50cebee-32bc-48a5-94cd-9978e48c02f5, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 017b1da4-7c9b-477d-92a3-29b2248317d3 was re-scheduled: Binding failed for port f50cebee-32bc-48a5-94cd-9978e48c02f5, please check neutron logs for more information.\n'] [ 648.509533] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 648.509769] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 017b1da4-7c9b-477d-92a3-29b2248317d3.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 017b1da4-7c9b-477d-92a3-29b2248317d3. [ 648.510087] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 017b1da4-7c9b-477d-92a3-29b2248317d3] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 017b1da4-7c9b-477d-92a3-29b2248317d3. [ 648.536477] nova-conductor[53688]: DEBUG nova.network.neutron [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 017b1da4-7c9b-477d-92a3-29b2248317d3] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 648.553097] nova-conductor[53688]: DEBUG nova.network.neutron [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 017b1da4-7c9b-477d-92a3-29b2248317d3] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 648.561245] nova-conductor[53688]: DEBUG nova.network.neutron [None req-cce9a24c-b562-446f-81d0-0af97fd3dc3f tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 017b1da4-7c9b-477d-92a3-29b2248317d3] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 649.612560] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-1b694e26-06b6-41e7-8a6d-599487fba9b1 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Expecting reply to msg c7963102e7954ef381d08411fd3c2bb5 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 649.754309] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg c7963102e7954ef381d08411fd3c2bb5 [ 650.115078] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-1b694e26-06b6-41e7-8a6d-599487fba9b1 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 650.121730] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-1b694e26-06b6-41e7-8a6d-599487fba9b1 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 46488b4f-4776-44bd-aa16-0d769c537e4d] While scheduling instance, the build request was already deleted. {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1712}} [ 650.136687] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Expecting reply to msg df27827b17f94947925b1c004a3fa7ac in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 650.267007] nova-conductor[53688]: INFO nova.scheduler.client.report [None req-1b694e26-06b6-41e7-8a6d-599487fba9b1 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Deleted allocations for instance 46488b4f-4776-44bd-aa16-0d769c537e4d [ 650.271157] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg df27827b17f94947925b1c004a3fa7ac [ 650.273687] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1b694e26-06b6-41e7-8a6d-599487fba9b1 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 650.273912] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1b694e26-06b6-41e7-8a6d-599487fba9b1 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 650.274081] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1b694e26-06b6-41e7-8a6d-599487fba9b1 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 650.274436] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1b694e26-06b6-41e7-8a6d-599487fba9b1 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 650.274619] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1b694e26-06b6-41e7-8a6d-599487fba9b1 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 650.275047] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-1b694e26-06b6-41e7-8a6d-599487fba9b1 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 650.286711] nova-conductor[53688]: DEBUG nova.quota [None req-1b694e26-06b6-41e7-8a6d-599487fba9b1 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Getting quotas for project c06b50a7aaa742afbbd0c6fc56c3d131. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 650.288838] nova-conductor[53688]: DEBUG nova.quota [None req-1b694e26-06b6-41e7-8a6d-599487fba9b1 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Getting quotas for user 3078a2af81b248f8b100f58ee66a5a86 and project c06b50a7aaa742afbbd0c6fc56c3d131. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 650.641381] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 650.656697] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 650.656697] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 650.656789] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 650.690767] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 650.691178] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 650.691511] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 650.692274] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 650.692594] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 650.692899] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 650.700976] nova-conductor[53689]: DEBUG nova.quota [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Getting quotas for project c06b50a7aaa742afbbd0c6fc56c3d131. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 650.703185] nova-conductor[53689]: DEBUG nova.quota [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Getting quotas for user 3078a2af81b248f8b100f58ee66a5a86 and project c06b50a7aaa742afbbd0c6fc56c3d131. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 650.708540] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: c376b161-74f9-405a-bb86-516583a9a76f] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 650.709134] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 650.709433] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 650.709694] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 650.714972] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: c376b161-74f9-405a-bb86-516583a9a76f] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 650.715851] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 650.716195] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 650.716464] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 650.735128] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 650.735356] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 650.735522] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 652.876379] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] [instance: 32b57d1b-d35f-488e-be23-9119f2f56562] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 35e64d9a-ac71-4a0f-abb8-c800619dff5a, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 32b57d1b-d35f-488e-be23-9119f2f56562 was re-scheduled: Binding failed for port 35e64d9a-ac71-4a0f-abb8-c800619dff5a, please check neutron logs for more information.\n'] [ 652.877031] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 652.877265] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 32b57d1b-d35f-488e-be23-9119f2f56562.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 32b57d1b-d35f-488e-be23-9119f2f56562. [ 652.877576] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] [instance: 32b57d1b-d35f-488e-be23-9119f2f56562] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 32b57d1b-d35f-488e-be23-9119f2f56562. [ 652.901134] nova-conductor[53689]: DEBUG nova.network.neutron [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] [instance: 32b57d1b-d35f-488e-be23-9119f2f56562] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 652.923946] nova-conductor[53689]: DEBUG nova.network.neutron [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] [instance: 32b57d1b-d35f-488e-be23-9119f2f56562] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 652.927526] nova-conductor[53689]: DEBUG nova.network.neutron [None req-440251be-ae2c-4a35-a60c-febed580cfec tempest-ServerTagsTestJSON-148504296 tempest-ServerTagsTestJSON-148504296-project-member] [instance: 32b57d1b-d35f-488e-be23-9119f2f56562] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 656.515145] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Expecting reply to msg 0e95d051762a4ebea14b59664440e4fd in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 656.655105] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 0e95d051762a4ebea14b59664440e4fd [ 657.023122] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 657.052232] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 657.052496] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 657.052695] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 657.091934] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 657.092201] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 657.092383] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 657.092798] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 657.092993] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 657.093157] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 657.102635] nova-conductor[53688]: DEBUG nova.quota [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Getting quotas for project e827316ff2e24c8da010c8f9def69e17. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 657.105658] nova-conductor[53688]: DEBUG nova.quota [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Getting quotas for user 8709dac9d659415082bbe312b40535d7 and project e827316ff2e24c8da010c8f9def69e17. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 657.115369] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] [instance: f82fd962-56c4-4cc2-b1dd-c51e7a71a9fc] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 657.115890] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 657.116120] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 657.116298] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 657.122260] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] [instance: f82fd962-56c4-4cc2-b1dd-c51e7a71a9fc] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 657.122950] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 657.123182] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 657.123353] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 657.138803] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 657.139038] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 657.139195] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-4f38619e-3871-4328-9efd-61cd5d12c60d tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 657.362765] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] [instance: d4d3db12-8de6-4daf-a087-89bb043d1217] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port b6e7038c-dfd6-439e-846f-4aba6ca1318b, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance d4d3db12-8de6-4daf-a087-89bb043d1217 was re-scheduled: Binding failed for port b6e7038c-dfd6-439e-846f-4aba6ca1318b, please check neutron logs for more information.\n'] [ 657.363340] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 657.363562] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d4d3db12-8de6-4daf-a087-89bb043d1217.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d4d3db12-8de6-4daf-a087-89bb043d1217. [ 657.363769] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] [instance: d4d3db12-8de6-4daf-a087-89bb043d1217] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d4d3db12-8de6-4daf-a087-89bb043d1217. [ 657.386715] nova-conductor[53689]: DEBUG nova.network.neutron [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] [instance: d4d3db12-8de6-4daf-a087-89bb043d1217] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 657.402097] nova-conductor[53689]: DEBUG nova.network.neutron [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] [instance: d4d3db12-8de6-4daf-a087-89bb043d1217] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 657.404839] nova-conductor[53689]: DEBUG nova.network.neutron [None req-1b6c5538-9fc0-45d5-a776-c5974fa8862a tempest-DeleteServersAdminTestJSON-581952752 tempest-DeleteServersAdminTestJSON-581952752-project-member] [instance: d4d3db12-8de6-4daf-a087-89bb043d1217] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 659.368407] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] [instance: 13c1d417-4087-46ad-b513-fc3317995d18] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port c0184233-b80c-46b2-a4e8-f4582c2edc01, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 13c1d417-4087-46ad-b513-fc3317995d18 was re-scheduled: Binding failed for port c0184233-b80c-46b2-a4e8-f4582c2edc01, please check neutron logs for more information.\n'] [ 659.371680] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 659.371996] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 13c1d417-4087-46ad-b513-fc3317995d18.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 13c1d417-4087-46ad-b513-fc3317995d18. [ 659.372381] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] [instance: 13c1d417-4087-46ad-b513-fc3317995d18] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 13c1d417-4087-46ad-b513-fc3317995d18. [ 659.393247] nova-conductor[53688]: DEBUG nova.network.neutron [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] [instance: 13c1d417-4087-46ad-b513-fc3317995d18] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 659.408567] nova-conductor[53688]: DEBUG nova.network.neutron [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] [instance: 13c1d417-4087-46ad-b513-fc3317995d18] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 659.411317] nova-conductor[53688]: DEBUG nova.network.neutron [None req-8f1451f3-cc8e-49e0-a234-0e4f6bacfeea tempest-ServersTestJSON-1053663491 tempest-ServersTestJSON-1053663491-project-member] [instance: 13c1d417-4087-46ad-b513-fc3317995d18] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 660.697387] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell0:SELECT=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 660.718436] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=3,nova_cell1:INSERT=7 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 660.728300] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=6,nova_api:UPDATE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 660.731378] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=6,nova_api:DELETE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 661.426695] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] [instance: 782dc85a-56f4-4f03-8711-b78bbadb33ce] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 90378eb2-b59a-48b8-a577-33399d56d5c9, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 782dc85a-56f4-4f03-8711-b78bbadb33ce was re-scheduled: Binding failed for port 90378eb2-b59a-48b8-a577-33399d56d5c9, please check neutron logs for more information.\n'] [ 661.427256] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 661.427477] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 782dc85a-56f4-4f03-8711-b78bbadb33ce.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 782dc85a-56f4-4f03-8711-b78bbadb33ce. [ 661.427684] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] [instance: 782dc85a-56f4-4f03-8711-b78bbadb33ce] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 782dc85a-56f4-4f03-8711-b78bbadb33ce. [ 661.453311] nova-conductor[53688]: DEBUG nova.network.neutron [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] [instance: 782dc85a-56f4-4f03-8711-b78bbadb33ce] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 661.479167] nova-conductor[53688]: DEBUG nova.network.neutron [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] [instance: 782dc85a-56f4-4f03-8711-b78bbadb33ce] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 661.486071] nova-conductor[53688]: DEBUG nova.network.neutron [None req-cd1781c5-927a-467d-bd44-c41c08f12193 tempest-InstanceActionsNegativeTestJSON-2128650360 tempest-InstanceActionsNegativeTestJSON-2128650360-project-member] [instance: 782dc85a-56f4-4f03-8711-b78bbadb33ce] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 663.564166] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Expecting reply to msg 0d217f0e532147bbbd8d02e0c263ef04 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 663.748562] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 0d217f0e532147bbbd8d02e0c263ef04 [ 664.069118] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 664.090570] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 664.090810] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 664.090988] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 664.125602] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 664.125844] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 664.126020] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 664.126489] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 664.126681] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 664.126867] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 664.139760] nova-conductor[53688]: DEBUG nova.quota [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Getting quotas for project 060b316170404510b74c98245e0dc553. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 664.155434] nova-conductor[53688]: DEBUG nova.quota [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Getting quotas for user 4030d1d160204c7db3ef8c948ca18168 and project 060b316170404510b74c98245e0dc553. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 664.162710] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] [instance: 2cd160c6-98ac-44a7-831e-d0fa3a958b99] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 664.163280] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 664.163502] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 664.163674] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 664.168553] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] [instance: 2cd160c6-98ac-44a7-831e-d0fa3a958b99] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 664.169405] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 664.169606] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 664.169770] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 664.185849] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 664.186116] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 664.186228] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 667.222512] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Expecting reply to msg 55743a9365304151a352d814d94a3852 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 667.480899] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 55743a9365304151a352d814d94a3852 [ 667.727302] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Took 0.51 seconds to select destinations for 2 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 667.765584] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 667.765584] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 667.765584] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 667.811207] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 667.811443] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 667.811620] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 667.837942] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 667.838246] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 667.838465] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 667.838896] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 667.839134] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 667.839340] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 667.850818] nova-conductor[53688]: DEBUG nova.quota [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Getting quotas for project 70c9e6d7debd4d8e8cb7790975294a22. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 667.854069] nova-conductor[53688]: DEBUG nova.quota [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Getting quotas for user f09e88112924400db81a4fbe611482f0 and project 70c9e6d7debd4d8e8cb7790975294a22. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 667.859921] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: 82f77423-cee6-4a04-8463-cabe57cba9cf] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 667.860501] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 667.860750] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 667.860970] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 667.866493] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: 82f77423-cee6-4a04-8463-cabe57cba9cf] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 667.867225] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 667.867480] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 667.867694] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 667.889206] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 667.890006] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 667.890236] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 667.900144] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: a0a40c68-77e2-4152-ac2e-059f8f7a8f78] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 667.900636] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 667.900840] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 667.901002] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 667.903858] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: a0a40c68-77e2-4152-ac2e-059f8f7a8f78] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 667.904593] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 667.904792] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 667.904955] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 667.920080] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 667.920316] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 667.920485] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 676.235346] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] [instance: 1336becb-9691-490c-86ea-3bc70d13d7df] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 65fe8495-4b99-4083-8128-022b6c4de52b, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 1336becb-9691-490c-86ea-3bc70d13d7df was re-scheduled: Binding failed for port 65fe8495-4b99-4083-8128-022b6c4de52b, please check neutron logs for more information.\n'] [ 676.235931] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 676.236173] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1336becb-9691-490c-86ea-3bc70d13d7df.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1336becb-9691-490c-86ea-3bc70d13d7df. [ 676.236381] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] [instance: 1336becb-9691-490c-86ea-3bc70d13d7df] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1336becb-9691-490c-86ea-3bc70d13d7df. [ 676.255068] nova-conductor[53689]: DEBUG nova.network.neutron [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] [instance: 1336becb-9691-490c-86ea-3bc70d13d7df] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 676.275199] nova-conductor[53689]: DEBUG nova.network.neutron [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] [instance: 1336becb-9691-490c-86ea-3bc70d13d7df] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 676.278370] nova-conductor[53689]: DEBUG nova.network.neutron [None req-26dddff7-42fe-47e4-843c-43822c25f550 tempest-ServersAdminNegativeTestJSON-285345347 tempest-ServersAdminNegativeTestJSON-285345347-project-member] [instance: 1336becb-9691-490c-86ea-3bc70d13d7df] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 677.843468] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell0:SELECT=4 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 677.908016] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SELECT=12,nova_cell1:SAVEPOINT=3,nova_cell1:INSERT=31,nova_cell1:RELEASE=3 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 677.913885] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=29,nova_api:UPDATE=2,nova_api:DELETE=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 677.916181] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=20,nova_api:DELETE=4,nova_api:UPDATE=2 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 678.201171] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] [instance: 30ed4438-4f74-4bc3-a6cc-a59420751940] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 17c48dca-cfb0-4cdc-9269-490a11efa464, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 30ed4438-4f74-4bc3-a6cc-a59420751940 was re-scheduled: Binding failed for port 17c48dca-cfb0-4cdc-9269-490a11efa464, please check neutron logs for more information.\n'] [ 678.201723] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 678.201967] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 30ed4438-4f74-4bc3-a6cc-a59420751940.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 30ed4438-4f74-4bc3-a6cc-a59420751940. [ 678.202289] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] [instance: 30ed4438-4f74-4bc3-a6cc-a59420751940] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 30ed4438-4f74-4bc3-a6cc-a59420751940. [ 678.226745] nova-conductor[53688]: DEBUG nova.network.neutron [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] [instance: 30ed4438-4f74-4bc3-a6cc-a59420751940] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 678.254388] nova-conductor[53688]: DEBUG nova.network.neutron [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] [instance: 30ed4438-4f74-4bc3-a6cc-a59420751940] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 678.257952] nova-conductor[53688]: DEBUG nova.network.neutron [None req-9857f842-36a2-4960-9ce2-230cb2785401 tempest-ServersTestManualDisk-2053152108 tempest-ServersTestManualDisk-2053152108-project-member] [instance: 30ed4438-4f74-4bc3-a6cc-a59420751940] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 680.191606] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] [instance: 767d7956-954b-4be7-8cc6-45872ff4cfce] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 7cf3ed88-f9cd-4e91-8812-0a8b7f548c5c, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 767d7956-954b-4be7-8cc6-45872ff4cfce was re-scheduled: Binding failed for port 7cf3ed88-f9cd-4e91-8812-0a8b7f548c5c, please check neutron logs for more information.\n'] [ 680.192241] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 680.192470] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 767d7956-954b-4be7-8cc6-45872ff4cfce.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 767d7956-954b-4be7-8cc6-45872ff4cfce. [ 680.192710] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] [instance: 767d7956-954b-4be7-8cc6-45872ff4cfce] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 767d7956-954b-4be7-8cc6-45872ff4cfce. [ 680.223760] nova-conductor[53689]: DEBUG nova.network.neutron [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] [instance: 767d7956-954b-4be7-8cc6-45872ff4cfce] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 680.286965] nova-conductor[53689]: DEBUG nova.network.neutron [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] [instance: 767d7956-954b-4be7-8cc6-45872ff4cfce] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 680.290997] nova-conductor[53689]: DEBUG nova.network.neutron [None req-117282fb-6d7f-43c4-ac08-bdd3abb18aff tempest-ServerActionsTestOtherB-1209735886 tempest-ServerActionsTestOtherB-1209735886-project-member] [instance: 767d7956-954b-4be7-8cc6-45872ff4cfce] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 683.445310] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Expecting reply to msg 52f57c7eb05845949bacfe368bdee448 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 683.612058] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 52f57c7eb05845949bacfe368bdee448 [ 683.948165] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 683.969759] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 683.970029] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 683.970222] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 684.012036] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 684.012278] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 684.012447] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 684.012828] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 684.013009] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 684.013166] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 684.029243] nova-conductor[53689]: DEBUG nova.quota [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Getting quotas for project ebdbfbef269c4ebcb1c5604a5f93268d. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 684.029952] nova-conductor[53689]: DEBUG nova.quota [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Getting quotas for user b718b811588443bfbed219e81ebb2c23 and project ebdbfbef269c4ebcb1c5604a5f93268d. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 684.037021] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] [instance: e62c0b97-cfa7-4acf-bdc5-93d6996c7806] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 684.037566] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 684.037778] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 684.037976] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 684.060589] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] [instance: e62c0b97-cfa7-4acf-bdc5-93d6996c7806] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 684.061245] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 684.061387] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 684.061561] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 684.079711] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 684.079939] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 684.080153] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 687.970346] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Expecting reply to msg 97b4e4fcff184311b45bcf3230cc7641 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 688.117020] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 97b4e4fcff184311b45bcf3230cc7641 [ 688.472783] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 688.485960] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 688.486191] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 688.486362] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 688.514224] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 688.514424] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 688.514572] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 688.514967] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 688.515155] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 688.515317] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 688.522576] nova-conductor[53689]: DEBUG nova.quota [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Getting quotas for project b6946d58a1af45eb96597fef8e0c62f5. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 688.531075] nova-conductor[53689]: DEBUG nova.quota [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Getting quotas for user 2aafa17161df48feb736cef1e2ab52fd and project b6946d58a1af45eb96597fef8e0c62f5. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 688.535841] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] [instance: 7bfdc548-4f10-4525-9ea1-3781f90ca81d] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 688.536317] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 688.536519] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 688.536684] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 688.549884] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] [instance: 7bfdc548-4f10-4525-9ea1-3781f90ca81d] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 688.550289] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 688.550490] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 688.550654] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 688.568944] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 688.569168] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 688.569352] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 689.620433] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] [instance: b41c6d21-5e7f-427f-95ce-830fe0da8bc6] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 7dd0ca68-2bda-4db1-8681-659736235fa8, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance b41c6d21-5e7f-427f-95ce-830fe0da8bc6 was re-scheduled: Binding failed for port 7dd0ca68-2bda-4db1-8681-659736235fa8, please check neutron logs for more information.\n'] [ 689.620433] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 689.620433] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance b41c6d21-5e7f-427f-95ce-830fe0da8bc6.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance b41c6d21-5e7f-427f-95ce-830fe0da8bc6. [ 689.620433] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] [instance: b41c6d21-5e7f-427f-95ce-830fe0da8bc6] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance b41c6d21-5e7f-427f-95ce-830fe0da8bc6. [ 689.657021] nova-conductor[53688]: DEBUG nova.network.neutron [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] [instance: b41c6d21-5e7f-427f-95ce-830fe0da8bc6] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 689.674068] nova-conductor[53688]: DEBUG nova.network.neutron [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] [instance: b41c6d21-5e7f-427f-95ce-830fe0da8bc6] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 689.678926] nova-conductor[53688]: DEBUG nova.network.neutron [None req-70a0794c-8d68-4920-918c-537e6e9bbf23 tempest-TenantUsagesTestJSON-1072048346 tempest-TenantUsagesTestJSON-1072048346-project-member] [instance: b41c6d21-5e7f-427f-95ce-830fe0da8bc6] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 690.352487] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SAVEPOINT=24,nova_cell1:RELEASE=24,nova_cell1:SELECT=312,nova_cell1:UPDATE=148,nova_cell1:INSERT=26 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 690.866940] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=322,nova_cell1:UPDATE=154,nova_cell1:INSERT=18,nova_cell1:SAVEPOINT=37,nova_cell1:RELEASE=37 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 695.251549] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Expecting reply to msg e0ba7e2991e8431bacc410c99d9c1925 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 695.410564] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg e0ba7e2991e8431bacc410c99d9c1925 [ 695.757397] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 695.824988] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 695.825225] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 695.825520] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 695.893784] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 695.894612] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 695.894834] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 695.895220] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 695.895410] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 695.895588] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 695.922270] nova-conductor[53688]: DEBUG nova.quota [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Getting quotas for project 35609af3eb7c487aa755932083286314. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 695.926218] nova-conductor[53688]: DEBUG nova.quota [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Getting quotas for user 30aef3fc075c43bdae873c542998ccd0 and project 35609af3eb7c487aa755932083286314. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 695.932358] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] [instance: 0b400fe1-d0d0-4820-9f56-56ccbad5465a] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 695.932812] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 695.933010] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 695.933174] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 695.937762] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] [instance: 0b400fe1-d0d0-4820-9f56-56ccbad5465a] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 695.938430] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 695.938634] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 695.938794] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 695.956955] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 695.957178] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 695.957352] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 698.503884] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Expecting reply to msg 9b88a8dfbb8d4892b9227c4b29d707ce in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 698.519177] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell0:SELECT=2 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 698.554168] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=6,nova_cell1:SAVEPOINT=2,nova_cell1:INSERT=16,nova_cell1:RELEASE=2 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 698.566096] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=12,nova_api:UPDATE=2,nova_api:DELETE=2 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 698.656962] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 9b88a8dfbb8d4892b9227c4b29d707ce [ 699.010776] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 699.028081] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 699.028884] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 699.029425] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 699.090853] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 699.091076] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 699.091284] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 699.091718] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 699.092158] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 699.092650] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 699.102064] nova-conductor[53689]: DEBUG nova.quota [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Getting quotas for project 1463a6a43bbe48f886efed86e8e24482. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 699.105245] nova-conductor[53689]: DEBUG nova.quota [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Getting quotas for user 9aff8d5e22844391837fad86f2cd243a and project 1463a6a43bbe48f886efed86e8e24482. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 699.110245] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] [instance: e6108eed-93b4-40a5-a61b-67aa5bbe2fda] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 699.110685] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 699.110891] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 699.111054] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 699.119789] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] [instance: e6108eed-93b4-40a5-a61b-67aa5bbe2fda] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 699.120559] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 699.120860] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 699.121137] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 699.139078] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 699.139309] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 699.139480] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 705.901282] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell0:SELECT=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 705.941088] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SELECT=3,nova_cell1:SAVEPOINT=1,nova_cell1:INSERT=8,nova_cell1:RELEASE=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 705.949520] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=5,nova_api:UPDATE=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 705.954124] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=7,nova_api:DELETE=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 708.428764] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] [instance: 12c47e99-faf4-4083-a46f-4e33c451e980] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 967f4a13-789b-471b-ab8f-72b7480a5afc, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 12c47e99-faf4-4083-a46f-4e33c451e980 was re-scheduled: Binding failed for port 967f4a13-789b-471b-ab8f-72b7480a5afc, please check neutron logs for more information.\n'] [ 708.429434] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 708.429559] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 12c47e99-faf4-4083-a46f-4e33c451e980.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 12c47e99-faf4-4083-a46f-4e33c451e980. [ 708.429905] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] [instance: 12c47e99-faf4-4083-a46f-4e33c451e980] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 12c47e99-faf4-4083-a46f-4e33c451e980. [ 708.451008] nova-conductor[53688]: DEBUG nova.network.neutron [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] [instance: 12c47e99-faf4-4083-a46f-4e33c451e980] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 708.465992] nova-conductor[53688]: DEBUG nova.network.neutron [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] [instance: 12c47e99-faf4-4083-a46f-4e33c451e980] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 708.469906] nova-conductor[53688]: DEBUG nova.network.neutron [None req-c6e2e821-28f8-4d2a-88df-ec57e567ef48 tempest-FloatingIPsAssociationTestJSON-885379146 tempest-FloatingIPsAssociationTestJSON-885379146-project-member] [instance: 12c47e99-faf4-4083-a46f-4e33c451e980] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 709.097095] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell0:SELECT=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 709.126051] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=3,nova_cell1:SAVEPOINT=1,nova_cell1:INSERT=8,nova_cell1:RELEASE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 709.134144] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=18,nova_api:UPDATE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 709.135576] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=7,nova_api:DELETE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 710.307167] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] [instance: 5ea287cd-ba85-446d-85d0-5a050fe49f17] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port d1633ad7-ab0c-43f3-bf09-b74926bf1ccf, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 5ea287cd-ba85-446d-85d0-5a050fe49f17 was re-scheduled: Binding failed for port d1633ad7-ab0c-43f3-bf09-b74926bf1ccf, please check neutron logs for more information.\n'] [ 710.307974] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 710.307974] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 5ea287cd-ba85-446d-85d0-5a050fe49f17.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 5ea287cd-ba85-446d-85d0-5a050fe49f17. [ 710.308439] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] [instance: 5ea287cd-ba85-446d-85d0-5a050fe49f17] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 5ea287cd-ba85-446d-85d0-5a050fe49f17. [ 710.342024] nova-conductor[53689]: DEBUG nova.network.neutron [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] [instance: 5ea287cd-ba85-446d-85d0-5a050fe49f17] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 710.357684] nova-conductor[53689]: DEBUG nova.network.neutron [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] [instance: 5ea287cd-ba85-446d-85d0-5a050fe49f17] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 710.361127] nova-conductor[53689]: DEBUG nova.network.neutron [None req-258cae5c-7377-4c04-81a0-3d6e8152b08a tempest-ServerActionsTestJSON-129923650 tempest-ServerActionsTestJSON-129923650-project-member] [instance: 5ea287cd-ba85-446d-85d0-5a050fe49f17] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 712.251781] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] [instance: 297a5546-6159-462c-a436-032d94855c00] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 230ecbbd-5bca-48b4-9f6f-7b581a49f268, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 297a5546-6159-462c-a436-032d94855c00 was re-scheduled: Binding failed for port 230ecbbd-5bca-48b4-9f6f-7b581a49f268, please check neutron logs for more information.\n'] [ 712.252392] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 712.252597] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 297a5546-6159-462c-a436-032d94855c00.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 297a5546-6159-462c-a436-032d94855c00. [ 712.252832] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] [instance: 297a5546-6159-462c-a436-032d94855c00] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 297a5546-6159-462c-a436-032d94855c00. [ 712.278616] nova-conductor[53689]: DEBUG nova.network.neutron [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] [instance: 297a5546-6159-462c-a436-032d94855c00] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 712.307198] nova-conductor[53689]: DEBUG nova.network.neutron [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] [instance: 297a5546-6159-462c-a436-032d94855c00] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 712.312605] nova-conductor[53689]: DEBUG nova.network.neutron [None req-5d4a7b16-5b36-4499-ab71-25e57615bc92 tempest-ServersNegativeTestJSON-1808660442 tempest-ServersNegativeTestJSON-1808660442-project-member] [instance: 297a5546-6159-462c-a436-032d94855c00] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 716.581509] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] [instance: 2b87dfbe-2b94-4787-a795-94f8b63f651c] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port ea7c2700-8347-4baa-b73d-d330af13f7ab, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 2b87dfbe-2b94-4787-a795-94f8b63f651c was re-scheduled: Binding failed for port ea7c2700-8347-4baa-b73d-d330af13f7ab, please check neutron logs for more information.\n'] [ 716.582159] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 716.582807] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 2b87dfbe-2b94-4787-a795-94f8b63f651c.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 2b87dfbe-2b94-4787-a795-94f8b63f651c. [ 716.583078] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] [instance: 2b87dfbe-2b94-4787-a795-94f8b63f651c] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 2b87dfbe-2b94-4787-a795-94f8b63f651c. [ 716.610373] nova-conductor[53689]: DEBUG nova.network.neutron [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] [instance: 2b87dfbe-2b94-4787-a795-94f8b63f651c] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 716.634951] nova-conductor[53689]: DEBUG nova.network.neutron [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] [instance: 2b87dfbe-2b94-4787-a795-94f8b63f651c] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 716.638274] nova-conductor[53689]: DEBUG nova.network.neutron [None req-1e27a361-db14-451a-b65f-741626e0d2e4 tempest-MigrationsAdminTest-1696452469 tempest-MigrationsAdminTest-1696452469-project-member] [instance: 2b87dfbe-2b94-4787-a795-94f8b63f651c] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 717.494794] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Expecting reply to msg dc8bd9bd91a04a9cab774cef5f425dc5 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 717.666537] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg dc8bd9bd91a04a9cab774cef5f425dc5 [ 717.924686] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Expecting reply to msg bf2e04c5e9304b88bd9854ffe7ed73be in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 718.009521] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Took 0.52 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 718.030214] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 718.030456] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 718.030609] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 718.071506] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 718.071642] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 718.071896] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 718.072235] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 718.072415] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 718.072583] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 718.084314] nova-conductor[53688]: DEBUG nova.quota [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Getting quotas for project 3051fa87e7e647c5a3d7bdd0e4c63778. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 718.086543] nova-conductor[53688]: DEBUG nova.quota [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Getting quotas for user 5cb53d9709f844fe9b79a83c6fd96729 and project 3051fa87e7e647c5a3d7bdd0e4c63778. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 718.091712] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] [instance: 05901bd4-2bad-405e-8e73-f6de4393a0f8] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 718.092201] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 718.092484] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 718.092671] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 718.098071] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] [instance: 05901bd4-2bad-405e-8e73-f6de4393a0f8] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 718.098770] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 718.098930] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 718.099279] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 718.111984] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg bf2e04c5e9304b88bd9854ffe7ed73be [ 718.119280] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 718.119379] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 718.119643] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 718.427555] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 718.443201] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 718.443535] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 718.443768] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 718.508359] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 718.508606] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 718.508773] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 718.509149] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 718.509375] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 718.509484] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 718.519197] nova-conductor[53689]: DEBUG nova.quota [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Getting quotas for project 3051fa87e7e647c5a3d7bdd0e4c63778. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 718.522741] nova-conductor[53689]: DEBUG nova.quota [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Getting quotas for user 5cb53d9709f844fe9b79a83c6fd96729 and project 3051fa87e7e647c5a3d7bdd0e4c63778. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 718.546073] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] [instance: e8e826d4-2463-41a7-8c63-fd9f47eceea6] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 718.546518] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 718.546729] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 718.546901] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 718.551590] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] [instance: e8e826d4-2463-41a7-8c63-fd9f47eceea6] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 718.552342] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 718.552557] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 718.552771] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 718.576143] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] [instance: e2560c8e-61c6-4343-82cb-47dc5b1997fb] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 9d5ceb62-1624-4a37-9b8b-42997a4fb38c, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance e2560c8e-61c6-4343-82cb-47dc5b1997fb was re-scheduled: Binding failed for port 9d5ceb62-1624-4a37-9b8b-42997a4fb38c, please check neutron logs for more information.\n'] [ 718.576904] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 718.577151] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance e2560c8e-61c6-4343-82cb-47dc5b1997fb.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance e2560c8e-61c6-4343-82cb-47dc5b1997fb. [ 718.577486] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] [instance: e2560c8e-61c6-4343-82cb-47dc5b1997fb] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance e2560c8e-61c6-4343-82cb-47dc5b1997fb. [ 718.604515] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 718.605439] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 718.605716] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 718.616907] nova-conductor[53688]: DEBUG nova.network.neutron [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] [instance: e2560c8e-61c6-4343-82cb-47dc5b1997fb] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 718.642381] nova-conductor[53688]: DEBUG nova.network.neutron [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] [instance: e2560c8e-61c6-4343-82cb-47dc5b1997fb] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 718.645558] nova-conductor[53688]: DEBUG nova.network.neutron [None req-4070bcf9-37dc-4465-b011-fc55e0fedc30 tempest-ServerAddressesTestJSON-1376018912 tempest-ServerAddressesTestJSON-1376018912-project-member] [instance: e2560c8e-61c6-4343-82cb-47dc5b1997fb] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 720.595288] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] [instance: 3635532a-2af3-4ef5-a922-37fc763c9708] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port cd73e761-b035-4816-a3f2-ac6f23681fc7, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 3635532a-2af3-4ef5-a922-37fc763c9708 was re-scheduled: Binding failed for port cd73e761-b035-4816-a3f2-ac6f23681fc7, please check neutron logs for more information.\n'] [ 720.595957] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 720.596217] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 3635532a-2af3-4ef5-a922-37fc763c9708.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 3635532a-2af3-4ef5-a922-37fc763c9708. [ 720.596435] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] [instance: 3635532a-2af3-4ef5-a922-37fc763c9708] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 3635532a-2af3-4ef5-a922-37fc763c9708. [ 720.625286] nova-conductor[53688]: DEBUG nova.network.neutron [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] [instance: 3635532a-2af3-4ef5-a922-37fc763c9708] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 720.643212] nova-conductor[53688]: DEBUG nova.network.neutron [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] [instance: 3635532a-2af3-4ef5-a922-37fc763c9708] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 720.645145] nova-conductor[53688]: DEBUG nova.network.neutron [None req-d26c7b29-1c09-4fcd-b378-a0fc6d64532f tempest-ServerGroupTestJSON-900251174 tempest-ServerGroupTestJSON-900251174-project-member] [instance: 3635532a-2af3-4ef5-a922-37fc763c9708] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 721.003034] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Expecting reply to msg 666ffd31c14b4e30bf14e5e69a7c09ff in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 721.175642] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 666ffd31c14b4e30bf14e5e69a7c09ff [ 721.505963] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 721.523175] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 721.523605] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 721.523605] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 721.556056] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 721.556289] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 721.556462] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 721.556833] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 721.557019] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 721.557178] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 721.569579] nova-conductor[53688]: DEBUG nova.quota [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Getting quotas for project c9d6bc18a1814fbb8d52af8d48eb4665. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 721.571329] nova-conductor[53688]: DEBUG nova.quota [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Getting quotas for user 239c4c941bf844c5b3c1883246738431 and project c9d6bc18a1814fbb8d52af8d48eb4665. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 721.576428] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] [instance: f6faf064-364d-4d24-9822-220bce47b3f0] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 721.576858] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 721.577070] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 721.577215] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 721.581916] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] [instance: f6faf064-364d-4d24-9822-220bce47b3f0] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 721.582605] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 721.582846] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 721.583062] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 721.604031] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 721.604031] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 721.604031] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 721.797147] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Expecting reply to msg 68e16ce664a4490899e4c1c99a21d949 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 721.976390] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 68e16ce664a4490899e4c1c99a21d949 [ 722.302193] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 722.318420] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 722.321618] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.003s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 722.322222] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 722.387688] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 722.387918] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 722.388251] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 722.388486] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 722.388677] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 722.388837] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 722.401111] nova-conductor[53688]: DEBUG nova.quota [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Getting quotas for project 14ea1bf75c4d449fb4d3a1d1d39d2777. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 722.404252] nova-conductor[53688]: DEBUG nova.quota [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Getting quotas for user 3e10b86ebedd4040ad77e02f68c20392 and project 14ea1bf75c4d449fb4d3a1d1d39d2777. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 722.409289] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] [instance: d63e20b1-e4ee-4c90-bc94-c4c05917fa1f] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 722.409890] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 722.410092] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 722.411128] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 722.415863] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] [instance: d63e20b1-e4ee-4c90-bc94-c4c05917fa1f] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 722.416595] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 722.416807] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 722.416957] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 722.460458] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 722.460720] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 722.460897] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 722.486972] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] [instance: 144c3c21-b18e-4997-a241-8ff21a3b4835] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port e4f545c6-0fe7-4d41-a153-b9d2b62b4dc4, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 144c3c21-b18e-4997-a241-8ff21a3b4835 was re-scheduled: Binding failed for port e4f545c6-0fe7-4d41-a153-b9d2b62b4dc4, please check neutron logs for more information.\n'] [ 722.487913] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 722.488401] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 144c3c21-b18e-4997-a241-8ff21a3b4835.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 144c3c21-b18e-4997-a241-8ff21a3b4835. [ 722.488927] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] [instance: 144c3c21-b18e-4997-a241-8ff21a3b4835] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 144c3c21-b18e-4997-a241-8ff21a3b4835. [ 722.516560] nova-conductor[53689]: DEBUG nova.network.neutron [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] [instance: 144c3c21-b18e-4997-a241-8ff21a3b4835] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 722.532482] nova-conductor[53689]: DEBUG nova.network.neutron [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] [instance: 144c3c21-b18e-4997-a241-8ff21a3b4835] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 722.535463] nova-conductor[53689]: DEBUG nova.network.neutron [None req-73d42730-9e06-4816-85bf-9175ea4cca99 tempest-ServerActionsV293TestJSON-1761589095 tempest-ServerActionsV293TestJSON-1761589095-project-member] [instance: 144c3c21-b18e-4997-a241-8ff21a3b4835] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 724.504226] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] [instance: bf40cc8c-4729-49c5-8c9d-e3ee09606aa5] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 2d7689b9-6f2a-4729-b658-80c4feb01295, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance bf40cc8c-4729-49c5-8c9d-e3ee09606aa5 was re-scheduled: Binding failed for port 2d7689b9-6f2a-4729-b658-80c4feb01295, please check neutron logs for more information.\n'] [ 724.504826] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 724.505098] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance bf40cc8c-4729-49c5-8c9d-e3ee09606aa5.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance bf40cc8c-4729-49c5-8c9d-e3ee09606aa5. [ 724.505471] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] [instance: bf40cc8c-4729-49c5-8c9d-e3ee09606aa5] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance bf40cc8c-4729-49c5-8c9d-e3ee09606aa5. [ 724.530937] nova-conductor[53688]: DEBUG nova.network.neutron [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] [instance: bf40cc8c-4729-49c5-8c9d-e3ee09606aa5] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 724.547341] nova-conductor[53688]: DEBUG nova.network.neutron [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] [instance: bf40cc8c-4729-49c5-8c9d-e3ee09606aa5] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 724.551866] nova-conductor[53688]: DEBUG nova.network.neutron [None req-98d95f9f-6622-4817-b1ed-60eccd2e5db5 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] [instance: bf40cc8c-4729-49c5-8c9d-e3ee09606aa5] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 725.708344] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Expecting reply to msg 647383e7bde0473184375bec6050cb01 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 725.870718] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 647383e7bde0473184375bec6050cb01 [ 726.206205] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 726.221441] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 726.221726] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 726.221905] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 726.253437] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 726.253670] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 726.253849] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 726.254293] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 726.254494] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 726.254659] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 726.265334] nova-conductor[53688]: DEBUG nova.quota [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Getting quotas for project c6ac2fa4041e4f24bee18db89143d7dc. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 726.268395] nova-conductor[53688]: DEBUG nova.quota [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Getting quotas for user 438a167fe61344bc9371e3b42d5344c8 and project c6ac2fa4041e4f24bee18db89143d7dc. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 726.273192] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] [instance: ebc03a5c-984f-4d58-abb0-da555adcfbac] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 726.273609] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 726.273831] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 726.274007] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 726.279163] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] [instance: ebc03a5c-984f-4d58-abb0-da555adcfbac] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 726.279633] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 726.279829] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 726.280052] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 726.297347] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 726.297999] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 726.297999] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 726.402048] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] [instance: 8d63e0a8-85a1-400b-a6f0-8e87c7945655] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 469e6b69-bd24-49e1-a6b3-a58c8e848eba, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 8d63e0a8-85a1-400b-a6f0-8e87c7945655 was re-scheduled: Binding failed for port 469e6b69-bd24-49e1-a6b3-a58c8e848eba, please check neutron logs for more information.\n'] [ 726.402737] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 726.403095] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 8d63e0a8-85a1-400b-a6f0-8e87c7945655.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 8d63e0a8-85a1-400b-a6f0-8e87c7945655. [ 726.403707] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] [instance: 8d63e0a8-85a1-400b-a6f0-8e87c7945655] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 8d63e0a8-85a1-400b-a6f0-8e87c7945655. [ 726.433434] nova-conductor[53688]: DEBUG nova.network.neutron [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] [instance: 8d63e0a8-85a1-400b-a6f0-8e87c7945655] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 726.453494] nova-conductor[53688]: DEBUG nova.network.neutron [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] [instance: 8d63e0a8-85a1-400b-a6f0-8e87c7945655] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 726.457569] nova-conductor[53688]: DEBUG nova.network.neutron [None req-f6fe3ed7-48ec-4806-b05c-9ce19e0fdfe6 tempest-ServersTestBootFromVolume-1266653635 tempest-ServersTestBootFromVolume-1266653635-project-member] [instance: 8d63e0a8-85a1-400b-a6f0-8e87c7945655] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 728.513721] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell0:SELECT=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 728.556557] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=3,nova_cell1:INSERT=7 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 728.594295] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=6,nova_api:UPDATE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 728.598223] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=6,nova_api:DELETE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 735.513030] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] [instance: 23774aa5-1608-495f-8015-29e25f856c69] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port f1a08c1d-bfd9-47a7-9221-49193552bcba, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 23774aa5-1608-495f-8015-29e25f856c69 was re-scheduled: Binding failed for port f1a08c1d-bfd9-47a7-9221-49193552bcba, please check neutron logs for more information.\n'] [ 735.514134] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 735.514486] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 23774aa5-1608-495f-8015-29e25f856c69.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 23774aa5-1608-495f-8015-29e25f856c69. [ 735.514851] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] [instance: 23774aa5-1608-495f-8015-29e25f856c69] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 23774aa5-1608-495f-8015-29e25f856c69. [ 735.540550] nova-conductor[53688]: DEBUG nova.network.neutron [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] [instance: 23774aa5-1608-495f-8015-29e25f856c69] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 735.556550] nova-conductor[53688]: DEBUG nova.network.neutron [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] [instance: 23774aa5-1608-495f-8015-29e25f856c69] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 735.560112] nova-conductor[53688]: DEBUG nova.network.neutron [None req-24c97928-8313-4706-8c5b-9b40234d869d tempest-ImagesOneServerTestJSON-1243653497 tempest-ImagesOneServerTestJSON-1243653497-project-member] [instance: 23774aa5-1608-495f-8015-29e25f856c69] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 736.261309] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell0:SELECT=4 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 736.282540] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SELECT=12,nova_cell1:SAVEPOINT=3,nova_cell1:INSERT=31,nova_cell1:RELEASE=3 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 736.290082] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=26,nova_api:UPDATE=4 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 736.293493] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=22,nova_api:DELETE=4 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 738.346313] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-1e71974d-aa8d-41ca-a251-05df644e93d2 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Targeting cell 84364d98-e6ce-480f-8cca-682929b0326f(cell1) for conductor method rebuild_instance {{(pid=53689) wrapper /opt/stack/nova/nova/conductor/manager.py:95}} [ 738.346545] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1e71974d-aa8d-41ca-a251-05df644e93d2 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 738.346736] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1e71974d-aa8d-41ca-a251-05df644e93d2 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 738.346894] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1e71974d-aa8d-41ca-a251-05df644e93d2 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 738.363217] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-1e71974d-aa8d-41ca-a251-05df644e93d2 tempest-ServerShowV247Test-937081600 tempest-ServerShowV247Test-937081600-project-member] [instance: 1efd9cfe-3a0c-412c-aa44-3bf650d08f9d] No migration record for the rebuild/evacuate request. {{(pid=53689) rebuild_instance /opt/stack/nova/nova/conductor/manager.py:1227}} [ 742.938052] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Expecting reply to msg 1c0b982f354f41e4972d1502a2764dea in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 743.091537] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 1c0b982f354f41e4972d1502a2764dea [ 743.441234] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 743.455950] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 743.456523] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 743.456900] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 743.504071] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 743.504390] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 743.504620] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 743.505059] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 743.505303] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 743.505525] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 743.513935] nova-conductor[53689]: DEBUG nova.quota [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Getting quotas for project 4a752424d76840dabab55a9202e7a635. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 743.516228] nova-conductor[53689]: DEBUG nova.quota [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Getting quotas for user 77bd4b765f214cb8bb602e8e52071531 and project 4a752424d76840dabab55a9202e7a635. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 743.521371] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] [instance: 7327b3d9-6f7e-4203-b77b-bc0271e3a6ab] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 743.521897] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 743.522178] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 743.522415] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 743.527305] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] [instance: 7327b3d9-6f7e-4203-b77b-bc0271e3a6ab] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 743.528167] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 743.528497] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 743.528674] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 743.543842] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 743.544259] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 743.544524] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 750.459685] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:UPDATE=191,nova_cell1:SELECT=371,nova_cell1:DELETE=2,nova_cell1:SAVEPOINT=37,nova_cell1:RELEASE=37,nova_cell1:INSERT=26 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 750.973775] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=376,nova_cell1:UPDATE=187,nova_cell1:INSERT=24,nova_cell1:SAVEPOINT=34,nova_cell1:RELEASE=34 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 752.002870] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] [instance: b6a158f8-6e2a-4967-ad05-761804ec6590] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 70e066e6-557a-491a-9f73-a77f76256833, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance b6a158f8-6e2a-4967-ad05-761804ec6590 was re-scheduled: Binding failed for port 70e066e6-557a-491a-9f73-a77f76256833, please check neutron logs for more information.\n'] [ 752.003431] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 752.003661] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance b6a158f8-6e2a-4967-ad05-761804ec6590.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance b6a158f8-6e2a-4967-ad05-761804ec6590. [ 752.004650] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] [instance: b6a158f8-6e2a-4967-ad05-761804ec6590] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance b6a158f8-6e2a-4967-ad05-761804ec6590. [ 752.028387] nova-conductor[53688]: DEBUG nova.network.neutron [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] [instance: b6a158f8-6e2a-4967-ad05-761804ec6590] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 752.043961] nova-conductor[53688]: DEBUG nova.network.neutron [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] [instance: b6a158f8-6e2a-4967-ad05-761804ec6590] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 752.055150] nova-conductor[53688]: DEBUG nova.network.neutron [None req-45f175ce-162c-47d2-9198-e74d4e7e13b3 tempest-InstanceActionsV221TestJSON-446942723 tempest-InstanceActionsV221TestJSON-446942723-project-member] [instance: b6a158f8-6e2a-4967-ad05-761804ec6590] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 753.511800] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell0:SELECT=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 753.531033] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=7,nova_cell1:UPDATE=3,nova_cell1:INSERT=9,nova_cell1:SAVEPOINT=1,nova_cell1:RELEASE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 753.538371] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=6 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 753.541377] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=8,nova_api:UPDATE=1,nova_api:DELETE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 754.710305] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-f0498efe-d98e-4ced-a3ef-1ecc80fe201a tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Targeting cell 84364d98-e6ce-480f-8cca-682929b0326f(cell1) for conductor method rebuild_instance {{(pid=53688) wrapper /opt/stack/nova/nova/conductor/manager.py:95}} [ 754.710524] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f0498efe-d98e-4ced-a3ef-1ecc80fe201a tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 754.710721] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f0498efe-d98e-4ced-a3ef-1ecc80fe201a tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 754.710879] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f0498efe-d98e-4ced-a3ef-1ecc80fe201a tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 754.724303] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-f0498efe-d98e-4ced-a3ef-1ecc80fe201a tempest-ServerShowV254Test-1951211204 tempest-ServerShowV254Test-1951211204-project-member] [instance: f82fd962-56c4-4cc2-b1dd-c51e7a71a9fc] No migration record for the rebuild/evacuate request. {{(pid=53688) rebuild_instance /opt/stack/nova/nova/conductor/manager.py:1227}} [ 759.302330] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Expecting reply to msg 8a2e37f0f7534b8680f547ae4d9bdbe9 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 759.436181] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 8a2e37f0f7534b8680f547ae4d9bdbe9 [ 759.805465] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 759.820210] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 759.820643] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 759.820882] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 759.869170] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 759.869514] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 759.869807] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 759.870381] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 759.870657] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 759.870920] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 759.886570] nova-conductor[53689]: DEBUG nova.quota [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Getting quotas for project 6b8d7d7387e44003b6b4bc488c7900f3. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 759.890480] nova-conductor[53689]: DEBUG nova.quota [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Getting quotas for user afd2b293ac5747749b0bae2b787277ca and project 6b8d7d7387e44003b6b4bc488c7900f3. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 759.896858] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] [instance: fcdd594c-b89f-4d0b-a4d5-2644b3b62b56] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 759.897694] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 759.897694] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 759.897819] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 759.903842] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] [instance: fcdd594c-b89f-4d0b-a4d5-2644b3b62b56] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 759.904813] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 759.905342] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 759.905791] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 759.924066] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 759.924532] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 759.924844] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 761.199305] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] [instance: 799d4a06-f7a3-4b92-8e96-ac076848fdd3] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 7eb3b801-5a05-42ee-8612-36ba45119473, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 799d4a06-f7a3-4b92-8e96-ac076848fdd3 was re-scheduled: Binding failed for port 7eb3b801-5a05-42ee-8612-36ba45119473, please check neutron logs for more information.\n'] [ 761.199855] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 761.200100] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 799d4a06-f7a3-4b92-8e96-ac076848fdd3.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 799d4a06-f7a3-4b92-8e96-ac076848fdd3. [ 761.200405] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] [instance: 799d4a06-f7a3-4b92-8e96-ac076848fdd3] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 799d4a06-f7a3-4b92-8e96-ac076848fdd3. [ 761.244970] nova-conductor[53689]: DEBUG nova.network.neutron [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] [instance: 799d4a06-f7a3-4b92-8e96-ac076848fdd3] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 761.265362] nova-conductor[53689]: DEBUG nova.network.neutron [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] [instance: 799d4a06-f7a3-4b92-8e96-ac076848fdd3] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 761.269058] nova-conductor[53689]: DEBUG nova.network.neutron [None req-227587c5-9ce8-4607-a942-f4ab390a602f tempest-SecurityGroupsTestJSON-905376977 tempest-SecurityGroupsTestJSON-905376977-project-member] [instance: 799d4a06-f7a3-4b92-8e96-ac076848fdd3] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 763.026416] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] [instance: 21213cff-55b3-48fd-91b4-6718f7819bc3] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port d0b95e54-3f33-48d6-9f74-c633d85b9772, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 21213cff-55b3-48fd-91b4-6718f7819bc3 was re-scheduled: Binding failed for port d0b95e54-3f33-48d6-9f74-c633d85b9772, please check neutron logs for more information.\n'] [ 763.027476] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 763.027476] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 21213cff-55b3-48fd-91b4-6718f7819bc3.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 21213cff-55b3-48fd-91b4-6718f7819bc3. [ 763.027476] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] [instance: 21213cff-55b3-48fd-91b4-6718f7819bc3] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 21213cff-55b3-48fd-91b4-6718f7819bc3. [ 763.051037] nova-conductor[53689]: DEBUG nova.network.neutron [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] [instance: 21213cff-55b3-48fd-91b4-6718f7819bc3] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 763.070001] nova-conductor[53689]: DEBUG nova.network.neutron [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] [instance: 21213cff-55b3-48fd-91b4-6718f7819bc3] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 763.073191] nova-conductor[53689]: DEBUG nova.network.neutron [None req-2157ae5d-80ef-4ab7-9582-3b72362f377b tempest-ServersV294TestFqdnHostnames-782809002 tempest-ServersV294TestFqdnHostnames-782809002-project-member] [instance: 21213cff-55b3-48fd-91b4-6718f7819bc3] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 764.731658] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=2 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 764.749019] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SELECT=4,nova_cell1:UPDATE=3,nova_cell1:INSERT=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 764.905777] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: f3df4816-ef02-4ecc-a8ca-4f0eaf286218] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 0a0b4bd2-3d3f-440c-98f7-c40827ffa1df, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance f3df4816-ef02-4ecc-a8ca-4f0eaf286218 was re-scheduled: Binding failed for port 0a0b4bd2-3d3f-440c-98f7-c40827ffa1df, please check neutron logs for more information.\n'] [ 764.906414] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 764.906580] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f3df4816-ef02-4ecc-a8ca-4f0eaf286218.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f3df4816-ef02-4ecc-a8ca-4f0eaf286218. [ 764.906796] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: f3df4816-ef02-4ecc-a8ca-4f0eaf286218] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f3df4816-ef02-4ecc-a8ca-4f0eaf286218. [ 764.945750] nova-conductor[53689]: DEBUG nova.network.neutron [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: f3df4816-ef02-4ecc-a8ca-4f0eaf286218] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 764.978589] nova-conductor[53689]: DEBUG nova.network.neutron [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: f3df4816-ef02-4ecc-a8ca-4f0eaf286218] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 764.983742] nova-conductor[53689]: DEBUG nova.network.neutron [None req-bcb5091e-96cb-49b0-b234-1e19130786d1 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: f3df4816-ef02-4ecc-a8ca-4f0eaf286218] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 766.098411] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Expecting reply to msg 4dfba139937445799d6cdbe5e658d779 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 766.240400] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 4dfba139937445799d6cdbe5e658d779 [ 766.607984] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 766.622349] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 766.622583] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 766.622778] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 766.665426] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 766.666012] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 766.666233] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 766.666777] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 766.667003] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 766.667193] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 766.677679] nova-conductor[53689]: DEBUG nova.quota [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Getting quotas for project a3de421e0f994df8b809ce0096753f23. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 766.680689] nova-conductor[53689]: DEBUG nova.quota [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Getting quotas for user c471664da5894985bf7478057ea19b73 and project a3de421e0f994df8b809ce0096753f23. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 766.685644] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: 6494039f-3716-4174-92c0-15df384e0878] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 766.686100] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 766.686343] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 766.686537] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 766.692527] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: 6494039f-3716-4174-92c0-15df384e0878] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 766.693390] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 766.693713] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 766.694065] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 766.713256] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 766.713479] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 766.713649] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 766.738803] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Expecting reply to msg 03a1542fa4f94a20b6ba00bba7168715 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 766.901750] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 03a1542fa4f94a20b6ba00bba7168715 [ 767.172680] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Expecting reply to msg c030bca1e89747f9b4190460711e77d3 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 767.241495] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 767.256808] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 767.257036] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 767.257202] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 767.305660] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 767.305892] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 767.306811] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 767.307271] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 767.307481] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 767.307643] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 767.325926] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg c030bca1e89747f9b4190460711e77d3 [ 767.326695] nova-conductor[53688]: DEBUG nova.quota [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Getting quotas for project f39358f90b3f468bbf87a5719c76273d. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 767.329362] nova-conductor[53688]: DEBUG nova.quota [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Getting quotas for user 5a8248232ec34833b3e19f7dfd5d63f3 and project f39358f90b3f468bbf87a5719c76273d. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 767.339462] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] [instance: 9debd209-244f-472a-b9d6-cf63bba98839] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 767.339462] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 767.339462] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 767.339462] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 767.349765] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] [instance: 9debd209-244f-472a-b9d6-cf63bba98839] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 767.350680] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 767.350897] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 767.351064] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 767.370397] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 767.370604] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 767.370880] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 767.493307] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Expecting reply to msg 56ebccadd2fb4a0c85674d91819ecde6 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 767.627068] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 56ebccadd2fb4a0c85674d91819ecde6 [ 767.677434] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 767.694371] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 767.694620] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 767.694802] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 767.714861] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 767.715071] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 767.715231] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 767.715586] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 767.715759] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 767.715912] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 767.723185] nova-conductor[53688]: DEBUG nova.quota [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Getting quotas for project f39358f90b3f468bbf87a5719c76273d. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 767.731150] nova-conductor[53688]: DEBUG nova.quota [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Getting quotas for user 5a8248232ec34833b3e19f7dfd5d63f3 and project f39358f90b3f468bbf87a5719c76273d. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 767.736320] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] [instance: faaabf1e-74af-4cfa-ba1c-e2c2fabad124] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 767.736885] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 767.736994] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 767.737161] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 767.740287] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] [instance: faaabf1e-74af-4cfa-ba1c-e2c2fabad124] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 767.740931] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 767.741580] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 767.741580] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 767.760063] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 767.760321] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 767.760508] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 767.805303] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] [instance: 2a7d4872-4ed7-4058-bc36-b199d89a9f14] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 8af34a72-d61f-4969-b3b5-acef24d3c087, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 2a7d4872-4ed7-4058-bc36-b199d89a9f14 was re-scheduled: Binding failed for port 8af34a72-d61f-4969-b3b5-acef24d3c087, please check neutron logs for more information.\n'] [ 767.805979] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 767.806250] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 2a7d4872-4ed7-4058-bc36-b199d89a9f14.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 2a7d4872-4ed7-4058-bc36-b199d89a9f14. [ 767.806610] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] [instance: 2a7d4872-4ed7-4058-bc36-b199d89a9f14] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 2a7d4872-4ed7-4058-bc36-b199d89a9f14. [ 767.839638] nova-conductor[53689]: DEBUG nova.network.neutron [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] [instance: 2a7d4872-4ed7-4058-bc36-b199d89a9f14] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 767.868500] nova-conductor[53689]: DEBUG nova.network.neutron [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] [instance: 2a7d4872-4ed7-4058-bc36-b199d89a9f14] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 767.872792] nova-conductor[53689]: DEBUG nova.network.neutron [None req-7e037969-2989-4a4a-bbff-66de7775f64d tempest-InstanceActionsTestJSON-1354842284 tempest-InstanceActionsTestJSON-1354842284-project-member] [instance: 2a7d4872-4ed7-4058-bc36-b199d89a9f14] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 767.998570] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 768.012817] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 768.013045] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 768.013214] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 768.045294] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 768.045529] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 768.045699] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 768.046138] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 768.046296] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 768.046452] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 768.056606] nova-conductor[53689]: DEBUG nova.quota [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Getting quotas for project f39358f90b3f468bbf87a5719c76273d. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 768.060054] nova-conductor[53689]: DEBUG nova.quota [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Getting quotas for user 5a8248232ec34833b3e19f7dfd5d63f3 and project f39358f90b3f468bbf87a5719c76273d. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 768.065385] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] [instance: 98f63a99-f1b8-4420-978d-7b69c39a2692] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 768.065749] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 768.065991] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 768.066158] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 768.070473] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] [instance: 98f63a99-f1b8-4420-978d-7b69c39a2692] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 768.071111] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 768.071311] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 768.071473] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 768.090014] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 768.090228] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 768.090426] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 769.070731] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Expecting reply to msg f57e4c2841aa4c15b31f31419bd2134e in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 769.325909] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg f57e4c2841aa4c15b31f31419bd2134e [ 769.573088] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Took 0.50 seconds to select destinations for 3 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 769.587258] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 769.587625] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 769.587809] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 769.630323] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 769.630557] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 769.630722] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 769.654928] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 769.655229] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 769.655398] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 769.689808] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 769.689947] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 769.690113] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 769.690874] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 769.691064] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 769.691228] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 769.701594] nova-conductor[53689]: DEBUG nova.quota [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Getting quotas for project f921325e5f6b410fa0137369ca3a9c89. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 769.705411] nova-conductor[53689]: DEBUG nova.quota [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Getting quotas for user 6fe5dcf103624f20b3e75d7f2bbd54b1 and project f921325e5f6b410fa0137369ca3a9c89. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 769.711350] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] [instance: 05df2575-9c3f-43d4-8fe4-52a808e11080] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 769.712416] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 769.712616] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 769.712803] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 769.719155] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] [instance: 05df2575-9c3f-43d4-8fe4-52a808e11080] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 769.720061] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 769.720503] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 769.720698] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 769.746253] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 769.746542] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 769.746738] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 769.756294] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] [instance: ae9866e2-544a-4d26-b198-87110f42f054] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 769.756904] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 769.757110] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 769.757338] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 769.761516] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] [instance: ae9866e2-544a-4d26-b198-87110f42f054] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 769.762464] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 769.762672] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 769.763494] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 769.791597] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 769.791867] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 769.792094] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 769.804571] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] [instance: b720f9f1-9401-40b1-978b-9b8eefe712ea] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 769.805069] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 769.805432] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 769.805600] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 769.878708] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] [instance: b720f9f1-9401-40b1-978b-9b8eefe712ea] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 769.879457] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 769.879702] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 769.880036] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 769.916304] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 769.916532] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 769.916699] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 777.719766] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell0:SELECT=2 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 777.743304] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SELECT=6,nova_cell1:SAVEPOINT=1,nova_cell1:INSERT=15,nova_cell1:RELEASE=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 777.754220] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=11,nova_api:UPDATE=1,nova_api:DELETE=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 777.756616] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=13,nova_api:UPDATE=1,nova_api:DELETE=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 778.092100] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: c376b161-74f9-405a-bb86-516583a9a76f] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 1798125d-2352-4879-a756-ef155a7b85e5, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance c376b161-74f9-405a-bb86-516583a9a76f was re-scheduled: Binding failed for port 1798125d-2352-4879-a756-ef155a7b85e5, please check neutron logs for more information.\n'] [ 778.092100] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 778.092454] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c376b161-74f9-405a-bb86-516583a9a76f.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c376b161-74f9-405a-bb86-516583a9a76f. [ 778.092508] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: c376b161-74f9-405a-bb86-516583a9a76f] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c376b161-74f9-405a-bb86-516583a9a76f. [ 778.120605] nova-conductor[53689]: DEBUG nova.network.neutron [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: c376b161-74f9-405a-bb86-516583a9a76f] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 778.138124] nova-conductor[53689]: DEBUG nova.network.neutron [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: c376b161-74f9-405a-bb86-516583a9a76f] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 778.144449] nova-conductor[53689]: DEBUG nova.network.neutron [None req-3802985d-bbb7-4539-94f4-f7f92ad3a1da tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: c376b161-74f9-405a-bb86-516583a9a76f] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 779.489312] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Expecting reply to msg 4248113c04d04ffb95c64e6a22f333a5 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 779.616098] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 4248113c04d04ffb95c64e6a22f333a5 [ 779.696297] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell0:SELECT=4 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 779.882673] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=16,nova_cell1:SAVEPOINT=2,nova_cell1:INSERT=44,nova_cell1:RELEASE=2 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 779.900131] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=32,nova_api:DELETE=3,nova_api:UPDATE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 779.908579] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=24,nova_api:UPDATE=5,nova_api:DELETE=3 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 779.991596] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 780.005195] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 780.005421] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 780.005590] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 780.031398] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 780.031667] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 780.031832] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 780.032222] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 780.032403] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 780.032554] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 780.042262] nova-conductor[53688]: DEBUG nova.quota [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Getting quotas for project c06b50a7aaa742afbbd0c6fc56c3d131. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 780.045257] nova-conductor[53688]: DEBUG nova.quota [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Getting quotas for user 3078a2af81b248f8b100f58ee66a5a86 and project c06b50a7aaa742afbbd0c6fc56c3d131. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 780.049959] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 9952d347-2ca7-48f2-8ee1-dc1d767402dc] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 780.050445] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 780.050664] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 780.050863] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 780.054588] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 9952d347-2ca7-48f2-8ee1-dc1d767402dc] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 780.055061] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 780.055262] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 780.055417] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 780.074419] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 780.074512] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 780.074680] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 785.563546] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] [instance: 2cd160c6-98ac-44a7-831e-d0fa3a958b99] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port b17ade9b-9979-4941-8442-f0ef91d65a14, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 2cd160c6-98ac-44a7-831e-d0fa3a958b99 was re-scheduled: Binding failed for port b17ade9b-9979-4941-8442-f0ef91d65a14, please check neutron logs for more information.\n'] [ 785.563546] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 785.563546] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 2cd160c6-98ac-44a7-831e-d0fa3a958b99.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 2cd160c6-98ac-44a7-831e-d0fa3a958b99. [ 785.563546] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] [instance: 2cd160c6-98ac-44a7-831e-d0fa3a958b99] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 2cd160c6-98ac-44a7-831e-d0fa3a958b99. [ 785.581054] nova-conductor[53688]: DEBUG nova.network.neutron [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] [instance: 2cd160c6-98ac-44a7-831e-d0fa3a958b99] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 785.597282] nova-conductor[53688]: DEBUG nova.network.neutron [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] [instance: 2cd160c6-98ac-44a7-831e-d0fa3a958b99] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 785.600674] nova-conductor[53688]: DEBUG nova.network.neutron [None req-e9427d7b-83ba-4396-a2f6-148e7d395fb2 tempest-ServerMetadataTestJSON-473901767 tempest-ServerMetadataTestJSON-473901767-project-member] [instance: 2cd160c6-98ac-44a7-831e-d0fa3a958b99] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 787.495798] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: 82f77423-cee6-4a04-8463-cabe57cba9cf] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 6d97bb17-557c-40e8-91c4-7a67086718c1, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 82f77423-cee6-4a04-8463-cabe57cba9cf was re-scheduled: Binding failed for port 6d97bb17-557c-40e8-91c4-7a67086718c1, please check neutron logs for more information.\n'] [ 787.496111] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 787.496250] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 82f77423-cee6-4a04-8463-cabe57cba9cf.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 82f77423-cee6-4a04-8463-cabe57cba9cf. [ 787.496657] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: 82f77423-cee6-4a04-8463-cabe57cba9cf] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 82f77423-cee6-4a04-8463-cabe57cba9cf. [ 787.527814] nova-conductor[53689]: DEBUG nova.network.neutron [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: 82f77423-cee6-4a04-8463-cabe57cba9cf] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 787.543617] nova-conductor[53689]: DEBUG nova.network.neutron [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: 82f77423-cee6-4a04-8463-cabe57cba9cf] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 787.547286] nova-conductor[53689]: DEBUG nova.network.neutron [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: 82f77423-cee6-4a04-8463-cabe57cba9cf] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 789.420212] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: a0a40c68-77e2-4152-ac2e-059f8f7a8f78] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 111bf606-b67a-4d6a-8de1-a66912dc3f30, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance a0a40c68-77e2-4152-ac2e-059f8f7a8f78 was re-scheduled: Binding failed for port 111bf606-b67a-4d6a-8de1-a66912dc3f30, please check neutron logs for more information.\n'] [ 789.420802] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 789.421076] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance a0a40c68-77e2-4152-ac2e-059f8f7a8f78.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance a0a40c68-77e2-4152-ac2e-059f8f7a8f78. [ 789.421294] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: a0a40c68-77e2-4152-ac2e-059f8f7a8f78] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance a0a40c68-77e2-4152-ac2e-059f8f7a8f78. [ 789.444913] nova-conductor[53688]: DEBUG nova.network.neutron [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: a0a40c68-77e2-4152-ac2e-059f8f7a8f78] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 789.480721] nova-conductor[53688]: DEBUG nova.network.neutron [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: a0a40c68-77e2-4152-ac2e-059f8f7a8f78] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 789.484558] nova-conductor[53688]: DEBUG nova.network.neutron [None req-b5aa5be3-e859-48c8-9d6b-1df3bb5ee1ab tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: a0a40c68-77e2-4152-ac2e-059f8f7a8f78] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 790.038960] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell0:SELECT=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 790.058239] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SELECT=3,nova_cell1:INSERT=7 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 790.069151] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=7 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 790.071901] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=5,nova_api:UPDATE=1,nova_api:DELETE=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 790.783382] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Expecting reply to msg 939d9b063a694299aeaea643509f910c in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 791.075146] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 939d9b063a694299aeaea643509f910c [ 791.287385] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Took 0.51 seconds to select destinations for 2 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 791.300623] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 791.300860] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 791.301075] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 791.351469] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 791.351799] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 791.351986] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 791.376300] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 791.376527] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 791.376696] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 791.378316] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 791.378539] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 791.378717] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 791.388408] nova-conductor[53689]: DEBUG nova.quota [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Getting quotas for project 70c9e6d7debd4d8e8cb7790975294a22. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 791.397201] nova-conductor[53689]: DEBUG nova.quota [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Getting quotas for user f09e88112924400db81a4fbe611482f0 and project 70c9e6d7debd4d8e8cb7790975294a22. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 791.402358] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: 1fde207b-9d32-4cff-b3fe-d0caddd20f69] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 791.402817] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 791.403013] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 791.403779] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 791.408051] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: 1fde207b-9d32-4cff-b3fe-d0caddd20f69] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 791.409833] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 791.410037] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 791.410197] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 791.444349] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 791.444349] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 791.444536] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 791.451078] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: 6182ea84-d5f2-4f01-9091-3d7b0b096d7c] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 791.451396] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 791.451601] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 791.451834] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 791.455124] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: 6182ea84-d5f2-4f01-9091-3d7b0b096d7c] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 791.455773] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 791.455965] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 791.456149] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 791.473821] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 791.474094] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 791.474278] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 791.704298] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Expecting reply to msg 58cfdf1f8acf4be39630d62bd19fd0f0 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 791.868091] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 58cfdf1f8acf4be39630d62bd19fd0f0 [ 792.207794] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 792.222716] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 792.222951] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 792.223134] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 792.281581] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 792.281847] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 792.282014] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 792.282422] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 792.282573] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 792.282770] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 792.295188] nova-conductor[53688]: DEBUG nova.quota [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Getting quotas for project d9325f1def284d2a9fdced4e9eeb17f0. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 792.298462] nova-conductor[53688]: DEBUG nova.quota [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Getting quotas for user af34c4e3d81c4729a9dd4a8531992ff1 and project d9325f1def284d2a9fdced4e9eeb17f0. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 792.305726] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 27e43d79-6435-46fb-ac71-9be7313d591a] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 792.306243] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 792.306313] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 792.306429] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 792.311665] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 27e43d79-6435-46fb-ac71-9be7313d591a] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 792.312345] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 792.312549] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 792.312834] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 792.335056] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 792.335216] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 792.335382] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 801.383414] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell0:SELECT=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 801.458913] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=5,nova_cell1:INSERT=14 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 801.467301] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=8,nova_api:UPDATE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 801.471877] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=8,nova_api:UPDATE=1,nova_api:DELETE=2 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 802.290322] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell0:SELECT=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 802.321821] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SELECT=3,nova_cell1:SAVEPOINT=1,nova_cell1:INSERT=8,nova_cell1:RELEASE=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 802.329473] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=4,nova_api:UPDATE=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 802.333224] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=8,nova_api:DELETE=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 808.378239] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] [instance: e62c0b97-cfa7-4acf-bdc5-93d6996c7806] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port bba752d6-1653-445d-af20-3f45b1d50fbc, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance e62c0b97-cfa7-4acf-bdc5-93d6996c7806 was re-scheduled: Binding failed for port bba752d6-1653-445d-af20-3f45b1d50fbc, please check neutron logs for more information.\n'] [ 808.378800] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 808.379008] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance e62c0b97-cfa7-4acf-bdc5-93d6996c7806.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance e62c0b97-cfa7-4acf-bdc5-93d6996c7806. [ 808.379339] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] [instance: e62c0b97-cfa7-4acf-bdc5-93d6996c7806] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance e62c0b97-cfa7-4acf-bdc5-93d6996c7806. [ 808.400370] nova-conductor[53689]: DEBUG nova.network.neutron [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] [instance: e62c0b97-cfa7-4acf-bdc5-93d6996c7806] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 808.415747] nova-conductor[53689]: DEBUG nova.network.neutron [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] [instance: e62c0b97-cfa7-4acf-bdc5-93d6996c7806] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 808.418564] nova-conductor[53689]: DEBUG nova.network.neutron [None req-f5fd2d51-640b-4554-9c67-19f076a954a4 tempest-AttachInterfacesUnderV243Test-1945262616 tempest-AttachInterfacesUnderV243Test-1945262616-project-member] [instance: e62c0b97-cfa7-4acf-bdc5-93d6996c7806] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 810.200133] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] [instance: 7bfdc548-4f10-4525-9ea1-3781f90ca81d] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port e8d4b3f1-4266-42ad-a1e2-e0967aa5c4c2, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 7bfdc548-4f10-4525-9ea1-3781f90ca81d was re-scheduled: Binding failed for port e8d4b3f1-4266-42ad-a1e2-e0967aa5c4c2, please check neutron logs for more information.\n'] [ 810.200133] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 810.200743] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 7bfdc548-4f10-4525-9ea1-3781f90ca81d.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 7bfdc548-4f10-4525-9ea1-3781f90ca81d. [ 810.201184] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] [instance: 7bfdc548-4f10-4525-9ea1-3781f90ca81d] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 7bfdc548-4f10-4525-9ea1-3781f90ca81d. [ 810.238657] nova-conductor[53689]: DEBUG nova.network.neutron [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] [instance: 7bfdc548-4f10-4525-9ea1-3781f90ca81d] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 810.314853] nova-conductor[53689]: DEBUG nova.network.neutron [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] [instance: 7bfdc548-4f10-4525-9ea1-3781f90ca81d] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 810.327173] nova-conductor[53689]: DEBUG nova.network.neutron [None req-c9249982-bf5d-45e3-9d8f-4571cb11aae0 tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] [instance: 7bfdc548-4f10-4525-9ea1-3781f90ca81d] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 810.709645] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SAVEPOINT=31,nova_cell1:RELEASE=31,nova_cell1:SELECT=340,nova_cell1:UPDATE=185,nova_cell1:INSERT=20,nova_cell1:DELETE=4 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 811.228146] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=358,nova_cell1:UPDATE=180,nova_cell1:INSERT=30,nova_cell1:SAVEPOINT=36,nova_cell1:RELEASE=36,nova_cell1:DELETE=3 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 814.241154] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Expecting reply to msg 5802f3fb6ebe4820826670920cc4627b in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 814.365574] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 5802f3fb6ebe4820826670920cc4627b [ 814.743591] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 814.757373] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 814.757508] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 814.757697] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 814.774216] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Expecting reply to msg bddacc77411c4636a311a623c763fb76 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 814.796015] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 814.796283] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 814.796457] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 814.797185] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 814.797456] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 814.797603] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 814.812316] nova-conductor[53689]: DEBUG nova.quota [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Getting quotas for project b6946d58a1af45eb96597fef8e0c62f5. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 814.818204] nova-conductor[53689]: DEBUG nova.quota [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Getting quotas for user 2aafa17161df48feb736cef1e2ab52fd and project b6946d58a1af45eb96597fef8e0c62f5. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 814.825127] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] [instance: 9adae455-b609-4ecb-8841-43fb4d826f84] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 814.825668] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 814.825888] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 814.826062] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 814.833231] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] [instance: 9adae455-b609-4ecb-8841-43fb4d826f84] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 814.834271] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 814.834444] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 814.834615] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 814.852271] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 814.852501] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 814.852665] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 814.926759] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg bddacc77411c4636a311a623c763fb76 [ 815.286128] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 815.297336] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 815.297697] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 815.297985] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 815.332814] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 815.333231] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 815.333709] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 815.334327] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 815.334759] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 815.335018] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 815.343390] nova-conductor[53689]: DEBUG nova.quota [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Getting quotas for project 63d2fb7de0ad453dbe6891e6974f1b66. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 815.346598] nova-conductor[53689]: DEBUG nova.quota [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Getting quotas for user 8969ac54b88a47028e5784f6575f2d61 and project 63d2fb7de0ad453dbe6891e6974f1b66. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 815.353163] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: 9cedc314-173e-4686-8ee5-28c2512dbcba] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 815.353719] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 815.354027] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 815.354389] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 815.358483] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: 9cedc314-173e-4686-8ee5-28c2512dbcba] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 815.359232] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 815.359583] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 815.359866] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 815.379291] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 815.379639] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 815.379933] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 819.277234] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] [instance: 0b400fe1-d0d0-4820-9f56-56ccbad5465a] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 602900da-fbec-4d44-a5ab-e4570ec93784, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 0b400fe1-d0d0-4820-9f56-56ccbad5465a was re-scheduled: Binding failed for port 602900da-fbec-4d44-a5ab-e4570ec93784, please check neutron logs for more information.\n'] [ 819.277863] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 819.278131] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 0b400fe1-d0d0-4820-9f56-56ccbad5465a.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 0b400fe1-d0d0-4820-9f56-56ccbad5465a. [ 819.278470] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] [instance: 0b400fe1-d0d0-4820-9f56-56ccbad5465a] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 0b400fe1-d0d0-4820-9f56-56ccbad5465a. [ 819.306852] nova-conductor[53688]: DEBUG nova.network.neutron [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] [instance: 0b400fe1-d0d0-4820-9f56-56ccbad5465a] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 819.323195] nova-conductor[53688]: DEBUG nova.network.neutron [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] [instance: 0b400fe1-d0d0-4820-9f56-56ccbad5465a] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 819.327041] nova-conductor[53688]: DEBUG nova.network.neutron [None req-5d9d0889-4107-43cf-a46d-69501800ab7f tempest-ServerAddressesNegativeTestJSON-1057268752 tempest-ServerAddressesNegativeTestJSON-1057268752-project-member] [instance: 0b400fe1-d0d0-4820-9f56-56ccbad5465a] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 821.125492] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] [instance: e6108eed-93b4-40a5-a61b-67aa5bbe2fda] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port eae1b954-6dd0-4d6c-b829-c5b225294270, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance e6108eed-93b4-40a5-a61b-67aa5bbe2fda was re-scheduled: Binding failed for port eae1b954-6dd0-4d6c-b829-c5b225294270, please check neutron logs for more information.\n'] [ 821.126119] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 821.126379] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance e6108eed-93b4-40a5-a61b-67aa5bbe2fda.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance e6108eed-93b4-40a5-a61b-67aa5bbe2fda. [ 821.126725] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] [instance: e6108eed-93b4-40a5-a61b-67aa5bbe2fda] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance e6108eed-93b4-40a5-a61b-67aa5bbe2fda. [ 821.157137] nova-conductor[53689]: DEBUG nova.network.neutron [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] [instance: e6108eed-93b4-40a5-a61b-67aa5bbe2fda] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 821.178949] nova-conductor[53689]: DEBUG nova.network.neutron [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] [instance: e6108eed-93b4-40a5-a61b-67aa5bbe2fda] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 821.189668] nova-conductor[53689]: DEBUG nova.network.neutron [None req-369926bc-5a2c-43b5-8bba-a7557c0d2358 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] [instance: e6108eed-93b4-40a5-a61b-67aa5bbe2fda] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 823.051378] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] [instance: 05901bd4-2bad-405e-8e73-f6de4393a0f8] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 86a1da90-c2b6-4b02-b3e4-60cb6f775b2c, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 05901bd4-2bad-405e-8e73-f6de4393a0f8 was re-scheduled: Binding failed for port 86a1da90-c2b6-4b02-b3e4-60cb6f775b2c, please check neutron logs for more information.\n'] [ 823.052067] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 823.052341] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 05901bd4-2bad-405e-8e73-f6de4393a0f8.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 05901bd4-2bad-405e-8e73-f6de4393a0f8. [ 823.052607] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] [instance: 05901bd4-2bad-405e-8e73-f6de4393a0f8] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 05901bd4-2bad-405e-8e73-f6de4393a0f8. [ 823.078741] nova-conductor[53689]: DEBUG nova.network.neutron [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] [instance: 05901bd4-2bad-405e-8e73-f6de4393a0f8] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 823.098149] nova-conductor[53689]: DEBUG nova.network.neutron [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] [instance: 05901bd4-2bad-405e-8e73-f6de4393a0f8] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 823.101325] nova-conductor[53689]: DEBUG nova.network.neutron [None req-f8be8826-1188-47f8-80e9-e95177b081f6 tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] [instance: 05901bd4-2bad-405e-8e73-f6de4393a0f8] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 823.850943] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Expecting reply to msg 45c710dbca5d494e9993ca4e2c9fc928 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 824.085425] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 45c710dbca5d494e9993ca4e2c9fc928 [ 824.354399] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 824.368350] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 824.368579] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 824.368745] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 824.400666] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 824.400893] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 824.401059] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 824.401419] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 824.401598] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 824.401785] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 824.410368] nova-conductor[53689]: DEBUG nova.quota [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Getting quotas for project 1463a6a43bbe48f886efed86e8e24482. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 824.413426] nova-conductor[53689]: DEBUG nova.quota [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Getting quotas for user 9aff8d5e22844391837fad86f2cd243a and project 1463a6a43bbe48f886efed86e8e24482. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 824.418251] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] [instance: 31ab5ebd-3df1-4e9f-bf53-69d47176da01] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 824.418657] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 824.418850] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 824.419012] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 824.423602] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] [instance: 31ab5ebd-3df1-4e9f-bf53-69d47176da01] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 824.424221] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 824.424417] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 824.424578] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 824.438252] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 824.438470] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 824.438605] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 824.985523] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] [instance: e8e826d4-2463-41a7-8c63-fd9f47eceea6] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 640780fc-ea34-42cf-b119-e6ca43151fad, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance e8e826d4-2463-41a7-8c63-fd9f47eceea6 was re-scheduled: Binding failed for port 640780fc-ea34-42cf-b119-e6ca43151fad, please check neutron logs for more information.\n'] [ 824.985523] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 824.985523] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance e8e826d4-2463-41a7-8c63-fd9f47eceea6.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance e8e826d4-2463-41a7-8c63-fd9f47eceea6. [ 824.987253] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] [instance: e8e826d4-2463-41a7-8c63-fd9f47eceea6] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance e8e826d4-2463-41a7-8c63-fd9f47eceea6. [ 825.009900] nova-conductor[53689]: DEBUG nova.network.neutron [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] [instance: e8e826d4-2463-41a7-8c63-fd9f47eceea6] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 825.024400] nova-conductor[53689]: DEBUG nova.network.neutron [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] [instance: e8e826d4-2463-41a7-8c63-fd9f47eceea6] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 825.027338] nova-conductor[53689]: DEBUG nova.network.neutron [None req-0150cb50-8c5d-44e5-9eb2-cfd5475cecfd tempest-ListImageFiltersTestJSON-2140387087 tempest-ListImageFiltersTestJSON-2140387087-project-member] [instance: e8e826d4-2463-41a7-8c63-fd9f47eceea6] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 826.726125] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] [instance: f6faf064-364d-4d24-9822-220bce47b3f0] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 34cf5a99-d31e-4c7d-aa80-312081d0f6cf, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance f6faf064-364d-4d24-9822-220bce47b3f0 was re-scheduled: Binding failed for port 34cf5a99-d31e-4c7d-aa80-312081d0f6cf, please check neutron logs for more information.\n'] [ 826.726688] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 826.726921] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f6faf064-364d-4d24-9822-220bce47b3f0.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f6faf064-364d-4d24-9822-220bce47b3f0. [ 826.727137] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] [instance: f6faf064-364d-4d24-9822-220bce47b3f0] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f6faf064-364d-4d24-9822-220bce47b3f0. [ 826.759292] nova-conductor[53689]: DEBUG nova.network.neutron [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] [instance: f6faf064-364d-4d24-9822-220bce47b3f0] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 826.781982] nova-conductor[53689]: DEBUG nova.network.neutron [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] [instance: f6faf064-364d-4d24-9822-220bce47b3f0] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 826.785602] nova-conductor[53689]: DEBUG nova.network.neutron [None req-fb7c379d-8f2e-45eb-99d0-1b012e6f3046 tempest-ServersNegativeTestMultiTenantJSON-392229405 tempest-ServersNegativeTestMultiTenantJSON-392229405-project-member] [instance: f6faf064-364d-4d24-9822-220bce47b3f0] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 829.768062] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Expecting reply to msg d889d08f9108478caea3eb3b22005c59 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 829.939812] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg d889d08f9108478caea3eb3b22005c59 [ 830.276493] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 830.288155] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 830.288576] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 830.289713] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 830.330710] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 830.331109] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 830.331607] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 830.332166] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 830.332607] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 830.332969] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 830.344779] nova-conductor[53689]: DEBUG nova.quota [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Getting quotas for project 3ec0b17396a7411eaf00c5abdba0a16d. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 830.348591] nova-conductor[53689]: DEBUG nova.quota [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Getting quotas for user 82cd1c91b7e04048bb5be1db3bdf6e2a and project 3ec0b17396a7411eaf00c5abdba0a16d. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 830.355748] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] [instance: 7dc5b4e9-15c3-4f47-9587-6d34bd44a5ef] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 830.356377] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 830.356705] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 830.357001] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 830.363023] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] [instance: 7dc5b4e9-15c3-4f47-9587-6d34bd44a5ef] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 830.363891] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 830.364251] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 830.364561] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 830.390570] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 830.390725] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 830.390965] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 833.429362] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] [instance: d63e20b1-e4ee-4c90-bc94-c4c05917fa1f] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 56e44824-11a2-4db8-8e08-d6d6b0ecf51d, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance d63e20b1-e4ee-4c90-bc94-c4c05917fa1f was re-scheduled: Binding failed for port 56e44824-11a2-4db8-8e08-d6d6b0ecf51d, please check neutron logs for more information.\n'] [ 833.429971] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 833.430196] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d63e20b1-e4ee-4c90-bc94-c4c05917fa1f.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d63e20b1-e4ee-4c90-bc94-c4c05917fa1f. [ 833.430518] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] [instance: d63e20b1-e4ee-4c90-bc94-c4c05917fa1f] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d63e20b1-e4ee-4c90-bc94-c4c05917fa1f. [ 833.471031] nova-conductor[53689]: DEBUG nova.network.neutron [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] [instance: d63e20b1-e4ee-4c90-bc94-c4c05917fa1f] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 833.486278] nova-conductor[53689]: DEBUG nova.network.neutron [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] [instance: d63e20b1-e4ee-4c90-bc94-c4c05917fa1f] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 833.489059] nova-conductor[53689]: DEBUG nova.network.neutron [None req-7a79b3f3-a343-4dd8-a252-83c0b573d07c tempest-ServerExternalEventsTest-1934076090 tempest-ServerExternalEventsTest-1934076090-project-member] [instance: d63e20b1-e4ee-4c90-bc94-c4c05917fa1f] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 835.267343] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] [instance: ebc03a5c-984f-4d58-abb0-da555adcfbac] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 33201922-1943-4bb7-92f4-71b9565e8b26, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance ebc03a5c-984f-4d58-abb0-da555adcfbac was re-scheduled: Binding failed for port 33201922-1943-4bb7-92f4-71b9565e8b26, please check neutron logs for more information.\n'] [ 835.267915] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 835.268168] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ebc03a5c-984f-4d58-abb0-da555adcfbac.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ebc03a5c-984f-4d58-abb0-da555adcfbac. [ 835.268382] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] [instance: ebc03a5c-984f-4d58-abb0-da555adcfbac] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ebc03a5c-984f-4d58-abb0-da555adcfbac. [ 835.294302] nova-conductor[53689]: DEBUG nova.network.neutron [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] [instance: ebc03a5c-984f-4d58-abb0-da555adcfbac] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 835.311456] nova-conductor[53689]: DEBUG nova.network.neutron [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] [instance: ebc03a5c-984f-4d58-abb0-da555adcfbac] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 835.315711] nova-conductor[53689]: DEBUG nova.network.neutron [None req-31d5c201-1a1c-4b46-b0ba-a40cb1ce0199 tempest-VolumesAdminNegativeTest-1865414953 tempest-VolumesAdminNegativeTest-1865414953-project-member] [instance: ebc03a5c-984f-4d58-abb0-da555adcfbac] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 837.158824] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] [instance: 7327b3d9-6f7e-4203-b77b-bc0271e3a6ab] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port ec48a1a2-a3b3-4ad6-9142-fd30912106b1, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 7327b3d9-6f7e-4203-b77b-bc0271e3a6ab was re-scheduled: Binding failed for port ec48a1a2-a3b3-4ad6-9142-fd30912106b1, please check neutron logs for more information.\n'] [ 837.159434] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 837.159699] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 7327b3d9-6f7e-4203-b77b-bc0271e3a6ab.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 7327b3d9-6f7e-4203-b77b-bc0271e3a6ab. [ 837.159927] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] [instance: 7327b3d9-6f7e-4203-b77b-bc0271e3a6ab] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 7327b3d9-6f7e-4203-b77b-bc0271e3a6ab. [ 837.204770] nova-conductor[53689]: DEBUG nova.network.neutron [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] [instance: 7327b3d9-6f7e-4203-b77b-bc0271e3a6ab] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 837.221910] nova-conductor[53689]: DEBUG nova.network.neutron [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] [instance: 7327b3d9-6f7e-4203-b77b-bc0271e3a6ab] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 837.225805] nova-conductor[53689]: DEBUG nova.network.neutron [None req-0beaa677-c31c-4323-8d74-cc333845720c tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] [instance: 7327b3d9-6f7e-4203-b77b-bc0271e3a6ab] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 838.324257] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Expecting reply to msg 1eaedc4d5df74ca7989b6fb8e9509a0c in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 838.478801] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 1eaedc4d5df74ca7989b6fb8e9509a0c [ 838.828762] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 838.842892] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 838.843156] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 838.843355] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 838.890369] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 838.890630] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 838.890804] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 838.891311] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 838.891404] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 838.891584] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 838.900943] nova-conductor[53689]: DEBUG nova.quota [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Getting quotas for project 5007ed6b2ee248ee9478fd6fe2d329d3. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 838.904391] nova-conductor[53689]: DEBUG nova.quota [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Getting quotas for user afebea38badd427c9eb0e0e73108283f and project 5007ed6b2ee248ee9478fd6fe2d329d3. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 838.910420] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] [instance: d3dafd33-91f8-481d-8f40-8c2e98a7587d] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 838.910872] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 838.911078] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 838.911241] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 838.924982] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] [instance: d3dafd33-91f8-481d-8f40-8c2e98a7587d] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 838.925847] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 838.926170] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 838.926404] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 838.970221] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 838.970461] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 838.970631] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 840.329335] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Expecting reply to msg 48d11621664a4323a7ee1d925045266d in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 840.466857] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 48d11621664a4323a7ee1d925045266d [ 840.832649] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 840.849463] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 840.849714] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 840.849887] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 840.904512] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 840.904746] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 840.904920] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 840.905290] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 840.905472] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 840.905631] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 840.919253] nova-conductor[53688]: DEBUG nova.quota [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Getting quotas for project 4a752424d76840dabab55a9202e7a635. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 840.923841] nova-conductor[53688]: DEBUG nova.quota [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Getting quotas for user 77bd4b765f214cb8bb602e8e52071531 and project 4a752424d76840dabab55a9202e7a635. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 840.930803] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] [instance: f9e23014-2fe0-4aab-b03c-8759dc1e5eb0] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 840.931260] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 840.931459] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 840.931623] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 840.945291] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] [instance: f9e23014-2fe0-4aab-b03c-8759dc1e5eb0] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 840.946016] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 840.946221] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 840.946385] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 840.987302] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 840.987531] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 840.987698] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 848.896266] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell0:SELECT=5 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 848.928462] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=15,nova_cell1:INSERT=38,nova_cell1:SAVEPOINT=3,nova_cell1:RELEASE=3 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 848.958360] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=26,nova_api:UPDATE=2,nova_api:DELETE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 848.964063] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=34,nova_api:UPDATE=3,nova_api:DELETE=4 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 850.373004] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] [instance: fcdd594c-b89f-4d0b-a4d5-2644b3b62b56] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 9a1e113b-fad2-4524-a3b5-29c7375c9242, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance fcdd594c-b89f-4d0b-a4d5-2644b3b62b56 was re-scheduled: Binding failed for port 9a1e113b-fad2-4524-a3b5-29c7375c9242, please check neutron logs for more information.\n'] [ 850.373377] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 850.373415] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance fcdd594c-b89f-4d0b-a4d5-2644b3b62b56.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance fcdd594c-b89f-4d0b-a4d5-2644b3b62b56. [ 850.373910] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] [instance: fcdd594c-b89f-4d0b-a4d5-2644b3b62b56] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance fcdd594c-b89f-4d0b-a4d5-2644b3b62b56. [ 850.395470] nova-conductor[53689]: DEBUG nova.network.neutron [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] [instance: fcdd594c-b89f-4d0b-a4d5-2644b3b62b56] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 850.475787] nova-conductor[53689]: DEBUG nova.network.neutron [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] [instance: fcdd594c-b89f-4d0b-a4d5-2644b3b62b56] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 850.478897] nova-conductor[53689]: DEBUG nova.network.neutron [None req-c03e671b-fa07-4498-8ea1-9c9117d2577f tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] [instance: fcdd594c-b89f-4d0b-a4d5-2644b3b62b56] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 850.910078] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell0:SELECT=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 850.949115] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SELECT=3,nova_cell1:INSERT=7 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 850.973964] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=5 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 850.983153] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=7,nova_api:UPDATE=1,nova_api:DELETE=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 852.173885] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: 6494039f-3716-4174-92c0-15df384e0878] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 2c33529b-82e4-4376-a0ac-52e2aa5b95c6, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 6494039f-3716-4174-92c0-15df384e0878 was re-scheduled: Binding failed for port 2c33529b-82e4-4376-a0ac-52e2aa5b95c6, please check neutron logs for more information.\n'] [ 852.173885] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 852.173885] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 6494039f-3716-4174-92c0-15df384e0878.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 6494039f-3716-4174-92c0-15df384e0878. [ 852.173885] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: 6494039f-3716-4174-92c0-15df384e0878] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 6494039f-3716-4174-92c0-15df384e0878. [ 852.212033] nova-conductor[53689]: DEBUG nova.network.neutron [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: 6494039f-3716-4174-92c0-15df384e0878] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 852.233297] nova-conductor[53689]: DEBUG nova.network.neutron [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: 6494039f-3716-4174-92c0-15df384e0878] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 852.243160] nova-conductor[53689]: DEBUG nova.network.neutron [None req-ce441822-9883-41c5-873d-f7db8e167cc0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: 6494039f-3716-4174-92c0-15df384e0878] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 853.038068] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Expecting reply to msg ae4a337381fa482ea19d8686afccaa67 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 853.182222] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg ae4a337381fa482ea19d8686afccaa67 [ 853.215329] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Expecting reply to msg 19c4d8dec98640e0a5bdb30bb3551e87 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 853.358439] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 19c4d8dec98640e0a5bdb30bb3551e87 [ 853.540619] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 853.555645] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 853.555852] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 853.556032] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 853.587786] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 853.588014] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 853.588190] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 853.588561] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 853.588743] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 853.588899] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 853.598055] nova-conductor[53689]: DEBUG nova.quota [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Getting quotas for project 6b8d7d7387e44003b6b4bc488c7900f3. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 853.601441] nova-conductor[53689]: DEBUG nova.quota [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Getting quotas for user afd2b293ac5747749b0bae2b787277ca and project 6b8d7d7387e44003b6b4bc488c7900f3. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 853.606166] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] [instance: d4a2025d-c128-45a2-b74c-a7fd2630d615] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 853.606553] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 853.606743] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 853.606903] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 853.612998] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] [instance: d4a2025d-c128-45a2-b74c-a7fd2630d615] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 853.613579] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 853.613770] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 853.613934] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 853.632717] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 853.632945] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 853.633118] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 853.718892] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 853.729033] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 853.729241] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 853.729404] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 853.749461] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 853.749677] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 853.749838] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 853.750183] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 853.750397] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 853.750556] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 853.757394] nova-conductor[53689]: DEBUG nova.quota [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Getting quotas for project a3de421e0f994df8b809ce0096753f23. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 853.759455] nova-conductor[53689]: DEBUG nova.quota [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Getting quotas for user c471664da5894985bf7478057ea19b73 and project a3de421e0f994df8b809ce0096753f23. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 853.764317] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: 1e7c4c50-1d33-4947-80bc-ac3ca2453fd5] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 853.764738] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 853.764929] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 853.765087] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 853.768678] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: 1e7c4c50-1d33-4947-80bc-ac3ca2453fd5] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 853.769254] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 853.769444] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 853.769612] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 853.782127] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 853.782323] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 853.782483] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 860.970792] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] [instance: 9debd209-244f-472a-b9d6-cf63bba98839] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 0f8ef003-3fad-4161-b909-098c989850aa, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 9debd209-244f-472a-b9d6-cf63bba98839 was re-scheduled: Binding failed for port 0f8ef003-3fad-4161-b909-098c989850aa, please check neutron logs for more information.\n'] [ 860.971766] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 860.972166] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9debd209-244f-472a-b9d6-cf63bba98839.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9debd209-244f-472a-b9d6-cf63bba98839. [ 860.973169] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] [instance: 9debd209-244f-472a-b9d6-cf63bba98839] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9debd209-244f-472a-b9d6-cf63bba98839. [ 861.000555] nova-conductor[53689]: DEBUG nova.network.neutron [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] [instance: 9debd209-244f-472a-b9d6-cf63bba98839] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 861.015573] nova-conductor[53689]: DEBUG nova.network.neutron [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] [instance: 9debd209-244f-472a-b9d6-cf63bba98839] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 861.018753] nova-conductor[53689]: DEBUG nova.network.neutron [None req-c4188ca8-21f0-4668-8f75-2059b2925a67 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] [instance: 9debd209-244f-472a-b9d6-cf63bba98839] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 862.722348] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] [instance: faaabf1e-74af-4cfa-ba1c-e2c2fabad124] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port ec4e0a91-b568-46e5-b1f8-d48f15b87d5c, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance faaabf1e-74af-4cfa-ba1c-e2c2fabad124 was re-scheduled: Binding failed for port ec4e0a91-b568-46e5-b1f8-d48f15b87d5c, please check neutron logs for more information.\n'] [ 862.722950] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 862.723199] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance faaabf1e-74af-4cfa-ba1c-e2c2fabad124.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance faaabf1e-74af-4cfa-ba1c-e2c2fabad124. [ 862.723525] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] [instance: faaabf1e-74af-4cfa-ba1c-e2c2fabad124] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance faaabf1e-74af-4cfa-ba1c-e2c2fabad124. [ 862.741932] nova-conductor[53688]: DEBUG nova.network.neutron [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] [instance: faaabf1e-74af-4cfa-ba1c-e2c2fabad124] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 862.803557] nova-conductor[53688]: DEBUG nova.network.neutron [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] [instance: faaabf1e-74af-4cfa-ba1c-e2c2fabad124] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 862.807450] nova-conductor[53688]: DEBUG nova.network.neutron [None req-c9eb77d9-4674-4b01-8c5c-0f447c5838bb tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] [instance: faaabf1e-74af-4cfa-ba1c-e2c2fabad124] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 863.753613] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell0:SELECT=2 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 863.771632] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=6,nova_cell1:INSERT=14 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 863.778320] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=13 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 863.780716] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=11,nova_api:UPDATE=2,nova_api:DELETE=2 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 864.504808] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] [instance: 98f63a99-f1b8-4420-978d-7b69c39a2692] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 5eb78d19-b6bf-451e-880f-0ae3168b6fd2, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 98f63a99-f1b8-4420-978d-7b69c39a2692 was re-scheduled: Binding failed for port 5eb78d19-b6bf-451e-880f-0ae3168b6fd2, please check neutron logs for more information.\n'] [ 864.505362] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 864.505589] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 98f63a99-f1b8-4420-978d-7b69c39a2692.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 98f63a99-f1b8-4420-978d-7b69c39a2692. [ 864.505797] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] [instance: 98f63a99-f1b8-4420-978d-7b69c39a2692] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 98f63a99-f1b8-4420-978d-7b69c39a2692. [ 864.524571] nova-conductor[53689]: DEBUG nova.network.neutron [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] [instance: 98f63a99-f1b8-4420-978d-7b69c39a2692] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 864.538926] nova-conductor[53689]: DEBUG nova.network.neutron [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] [instance: 98f63a99-f1b8-4420-978d-7b69c39a2692] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 864.541388] nova-conductor[53689]: DEBUG nova.network.neutron [None req-deaffb0e-96a1-4b52-8e20-4c6e8d077769 tempest-ListServerFiltersTestJSON-85515236 tempest-ListServerFiltersTestJSON-85515236-project-member] [instance: 98f63a99-f1b8-4420-978d-7b69c39a2692] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 866.351099] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] [instance: 05df2575-9c3f-43d4-8fe4-52a808e11080] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port d806ad66-e122-4622-8f56-bfe27013f645, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 05df2575-9c3f-43d4-8fe4-52a808e11080 was re-scheduled: Binding failed for port d806ad66-e122-4622-8f56-bfe27013f645, please check neutron logs for more information.\n'] [ 866.351099] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 866.351099] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 05df2575-9c3f-43d4-8fe4-52a808e11080.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 05df2575-9c3f-43d4-8fe4-52a808e11080. [ 866.351099] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] [instance: 05df2575-9c3f-43d4-8fe4-52a808e11080] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 05df2575-9c3f-43d4-8fe4-52a808e11080. [ 866.387380] nova-conductor[53688]: DEBUG nova.network.neutron [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] [instance: 05df2575-9c3f-43d4-8fe4-52a808e11080] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 866.404708] nova-conductor[53688]: DEBUG nova.network.neutron [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] [instance: 05df2575-9c3f-43d4-8fe4-52a808e11080] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 866.408763] nova-conductor[53688]: DEBUG nova.network.neutron [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] [instance: 05df2575-9c3f-43d4-8fe4-52a808e11080] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 868.123006] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] [instance: ae9866e2-544a-4d26-b198-87110f42f054] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port fdce2ef1-b287-49fc-9741-089b76b0ab0f, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance ae9866e2-544a-4d26-b198-87110f42f054 was re-scheduled: Binding failed for port fdce2ef1-b287-49fc-9741-089b76b0ab0f, please check neutron logs for more information.\n'] [ 868.123365] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 868.123365] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ae9866e2-544a-4d26-b198-87110f42f054.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ae9866e2-544a-4d26-b198-87110f42f054. [ 868.123547] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] [instance: ae9866e2-544a-4d26-b198-87110f42f054] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ae9866e2-544a-4d26-b198-87110f42f054. [ 868.149253] nova-conductor[53688]: DEBUG nova.network.neutron [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] [instance: ae9866e2-544a-4d26-b198-87110f42f054] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 868.322981] nova-conductor[53688]: DEBUG nova.network.neutron [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] [instance: ae9866e2-544a-4d26-b198-87110f42f054] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 868.324766] nova-conductor[53688]: DEBUG nova.network.neutron [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] [instance: ae9866e2-544a-4d26-b198-87110f42f054] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 871.152201] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:UPDATE=204,nova_cell1:INSERT=28,nova_cell1:SELECT=407,nova_cell1:SAVEPOINT=41,nova_cell1:RELEASE=41 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 871.299758] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=384,nova_cell1:UPDATE=184,nova_cell1:INSERT=30,nova_cell1:SAVEPOINT=45,nova_cell1:RELEASE=45 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 881.223634] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] [instance: b720f9f1-9401-40b1-978b-9b8eefe712ea] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port f0f1a201-d4ee-4abf-a2c8-f33f9a7b67f6, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance b720f9f1-9401-40b1-978b-9b8eefe712ea was re-scheduled: Binding failed for port f0f1a201-d4ee-4abf-a2c8-f33f9a7b67f6, please check neutron logs for more information.\n'] [ 881.224235] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 881.224458] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance b720f9f1-9401-40b1-978b-9b8eefe712ea.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance b720f9f1-9401-40b1-978b-9b8eefe712ea. [ 881.224668] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] [instance: b720f9f1-9401-40b1-978b-9b8eefe712ea] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance b720f9f1-9401-40b1-978b-9b8eefe712ea. [ 881.247984] nova-conductor[53689]: DEBUG nova.network.neutron [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] [instance: b720f9f1-9401-40b1-978b-9b8eefe712ea] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 881.262965] nova-conductor[53689]: DEBUG nova.network.neutron [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] [instance: b720f9f1-9401-40b1-978b-9b8eefe712ea] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 881.265955] nova-conductor[53689]: DEBUG nova.network.neutron [None req-959d6159-e58d-4679-8e3f-1badec2bc5fb tempest-ListServersNegativeTestJSON-1980913997 tempest-ListServersNegativeTestJSON-1980913997-project-member] [instance: b720f9f1-9401-40b1-978b-9b8eefe712ea] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 882.932535] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 9952d347-2ca7-48f2-8ee1-dc1d767402dc] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 94ad734d-164b-4d36-83a3-e69aa8777538, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 9952d347-2ca7-48f2-8ee1-dc1d767402dc was re-scheduled: Binding failed for port 94ad734d-164b-4d36-83a3-e69aa8777538, please check neutron logs for more information.\n'] [ 882.933114] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 882.933333] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9952d347-2ca7-48f2-8ee1-dc1d767402dc.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9952d347-2ca7-48f2-8ee1-dc1d767402dc. [ 882.933572] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 9952d347-2ca7-48f2-8ee1-dc1d767402dc] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9952d347-2ca7-48f2-8ee1-dc1d767402dc. [ 882.954667] nova-conductor[53688]: DEBUG nova.network.neutron [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 9952d347-2ca7-48f2-8ee1-dc1d767402dc] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 882.969194] nova-conductor[53688]: DEBUG nova.network.neutron [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 9952d347-2ca7-48f2-8ee1-dc1d767402dc] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 882.972332] nova-conductor[53688]: DEBUG nova.network.neutron [None req-87c9ed44-1ef0-4cc4-9e27-929d7440ed48 tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 9952d347-2ca7-48f2-8ee1-dc1d767402dc] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 884.676572] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: 1fde207b-9d32-4cff-b3fe-d0caddd20f69] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 700bf580-c47d-4fd9-8067-c5c7564d83b0, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 1fde207b-9d32-4cff-b3fe-d0caddd20f69 was re-scheduled: Binding failed for port 700bf580-c47d-4fd9-8067-c5c7564d83b0, please check neutron logs for more information.\n'] [ 884.677078] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 884.677307] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1fde207b-9d32-4cff-b3fe-d0caddd20f69.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1fde207b-9d32-4cff-b3fe-d0caddd20f69. [ 884.677511] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: 1fde207b-9d32-4cff-b3fe-d0caddd20f69] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1fde207b-9d32-4cff-b3fe-d0caddd20f69. [ 884.705989] nova-conductor[53688]: DEBUG nova.network.neutron [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: 1fde207b-9d32-4cff-b3fe-d0caddd20f69] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 884.736053] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Expecting reply to msg ceacfb98d28c4391b73ac6d56222733a in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 884.744348] nova-conductor[53688]: DEBUG nova.network.neutron [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: 1fde207b-9d32-4cff-b3fe-d0caddd20f69] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 884.753733] nova-conductor[53688]: DEBUG nova.network.neutron [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: 1fde207b-9d32-4cff-b3fe-d0caddd20f69] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 884.919576] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg ceacfb98d28c4391b73ac6d56222733a [ 885.238222] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 885.269542] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 885.269956] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 885.270364] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 885.327883] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 885.328146] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 885.328319] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 885.328714] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 885.328914] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 885.329073] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 885.340059] nova-conductor[53688]: DEBUG nova.quota [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Getting quotas for project c06b50a7aaa742afbbd0c6fc56c3d131. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 885.346625] nova-conductor[53688]: DEBUG nova.quota [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Getting quotas for user 3078a2af81b248f8b100f58ee66a5a86 and project c06b50a7aaa742afbbd0c6fc56c3d131. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 885.352832] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: a70f220a-fa34-44af-939f-29292b556897] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 885.353447] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 885.353648] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 885.353815] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 885.356842] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: a70f220a-fa34-44af-939f-29292b556897] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 885.357470] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 885.357661] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 885.357821] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 885.374087] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 885.374337] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 885.374507] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 888.999173] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Expecting reply to msg 4a7ae16b9b2e4c92961d02e32527d1f9 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 889.178794] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 4a7ae16b9b2e4c92961d02e32527d1f9 [ 889.504744] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 889.521252] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 889.521654] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 889.521932] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 889.556647] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 889.556871] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 889.557036] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 889.557388] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 889.557569] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 889.557725] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 889.566142] nova-conductor[53689]: DEBUG nova.quota [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Getting quotas for project a36723dc732b444e8831b049e9f804b8. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 889.568300] nova-conductor[53689]: DEBUG nova.quota [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Getting quotas for user 5345fbbca90446719473829e2ea02386 and project a36723dc732b444e8831b049e9f804b8. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 889.573987] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] [instance: c9214700-faf8-4a26-8084-ffe4a2c06480] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 889.574459] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 889.574610] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 889.574777] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 889.580487] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] [instance: c9214700-faf8-4a26-8084-ffe4a2c06480] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 889.581140] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 889.581339] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 889.581498] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 889.595278] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 889.595278] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 889.595278] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 895.333409] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell0:SELECT=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 895.359767] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SELECT=3,nova_cell1:INSERT=7 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 895.369910] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=8,nova_api:UPDATE=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 895.371085] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=4,nova_api:DELETE=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 897.765019] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: 6182ea84-d5f2-4f01-9091-3d7b0b096d7c] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 7fc6dd31-acd8-4eb7-952a-365a4a2614e1, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 6182ea84-d5f2-4f01-9091-3d7b0b096d7c was re-scheduled: Binding failed for port 7fc6dd31-acd8-4eb7-952a-365a4a2614e1, please check neutron logs for more information.\n'] [ 897.765380] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 897.765605] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 6182ea84-d5f2-4f01-9091-3d7b0b096d7c.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 6182ea84-d5f2-4f01-9091-3d7b0b096d7c. [ 897.765770] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: 6182ea84-d5f2-4f01-9091-3d7b0b096d7c] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 6182ea84-d5f2-4f01-9091-3d7b0b096d7c. [ 897.790644] nova-conductor[53688]: DEBUG nova.network.neutron [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: 6182ea84-d5f2-4f01-9091-3d7b0b096d7c] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 897.805871] nova-conductor[53688]: DEBUG nova.network.neutron [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: 6182ea84-d5f2-4f01-9091-3d7b0b096d7c] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 897.808651] nova-conductor[53688]: DEBUG nova.network.neutron [None req-825cd93b-58d0-432f-be98-06744830a079 tempest-MultipleCreateTestJSON-1846072068 tempest-MultipleCreateTestJSON-1846072068-project-member] [instance: 6182ea84-d5f2-4f01-9091-3d7b0b096d7c] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 899.468565] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 27e43d79-6435-46fb-ac71-9be7313d591a] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 038957b1-b320-43d4-bfde-f11c360acfb3, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 27e43d79-6435-46fb-ac71-9be7313d591a was re-scheduled: Binding failed for port 038957b1-b320-43d4-bfde-f11c360acfb3, please check neutron logs for more information.\n'] [ 899.469504] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 899.469725] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 27e43d79-6435-46fb-ac71-9be7313d591a.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 27e43d79-6435-46fb-ac71-9be7313d591a. [ 899.471724] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 27e43d79-6435-46fb-ac71-9be7313d591a] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 27e43d79-6435-46fb-ac71-9be7313d591a. [ 899.495709] nova-conductor[53688]: DEBUG nova.network.neutron [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 27e43d79-6435-46fb-ac71-9be7313d591a] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 899.552398] nova-conductor[53688]: DEBUG nova.network.neutron [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 27e43d79-6435-46fb-ac71-9be7313d591a] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 899.556864] nova-conductor[53688]: DEBUG nova.network.neutron [None req-f051d1e7-6a12-4ce0-b0a9-15b42a1b6c11 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 27e43d79-6435-46fb-ac71-9be7313d591a] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 899.562760] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell0:SELECT=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 899.587483] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=3,nova_cell1:SAVEPOINT=1,nova_cell1:INSERT=8,nova_cell1:RELEASE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 899.590824] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=6,nova_api:UPDATE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 899.595263] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=6,nova_api:DELETE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 901.059252] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-677692ba-fc6b-4f92-b345-a017fe146ab7 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Expecting reply to msg 8dd94d89c8e440c6b79eb414fc3035d2 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 901.183647] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 8dd94d89c8e440c6b79eb414fc3035d2 [ 901.460865] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Expecting reply to msg e26add8865324c0cab5fe8ebb3f31f15 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 901.561582] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-677692ba-fc6b-4f92-b345-a017fe146ab7 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 901.568975] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-677692ba-fc6b-4f92-b345-a017fe146ab7 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: b99cee58-5428-4171-bd7c-f15ec261b70b] While scheduling instance, the build request was already deleted. {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1712}} [ 901.572717] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg e26add8865324c0cab5fe8ebb3f31f15 [ 901.590147] nova-conductor[53689]: INFO nova.scheduler.client.report [None req-677692ba-fc6b-4f92-b345-a017fe146ab7 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Deleted allocations for instance b99cee58-5428-4171-bd7c-f15ec261b70b [ 901.597057] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-677692ba-fc6b-4f92-b345-a017fe146ab7 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 901.597283] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-677692ba-fc6b-4f92-b345-a017fe146ab7 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 901.597450] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-677692ba-fc6b-4f92-b345-a017fe146ab7 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 901.597805] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-677692ba-fc6b-4f92-b345-a017fe146ab7 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 901.597985] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-677692ba-fc6b-4f92-b345-a017fe146ab7 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 901.598139] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-677692ba-fc6b-4f92-b345-a017fe146ab7 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 901.605488] nova-conductor[53689]: DEBUG nova.quota [None req-677692ba-fc6b-4f92-b345-a017fe146ab7 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Getting quotas for project d9325f1def284d2a9fdced4e9eeb17f0. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 901.607851] nova-conductor[53689]: DEBUG nova.quota [None req-677692ba-fc6b-4f92-b345-a017fe146ab7 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Getting quotas for user af34c4e3d81c4729a9dd4a8531992ff1 and project d9325f1def284d2a9fdced4e9eeb17f0. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 901.963807] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 901.976989] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 901.977220] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 901.977387] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 902.012208] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 902.012208] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 902.012341] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 902.012693] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 902.012902] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 902.013058] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 902.021776] nova-conductor[53689]: DEBUG nova.quota [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Getting quotas for project d9325f1def284d2a9fdced4e9eeb17f0. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 902.023821] nova-conductor[53689]: DEBUG nova.quota [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Getting quotas for user af34c4e3d81c4729a9dd4a8531992ff1 and project d9325f1def284d2a9fdced4e9eeb17f0. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 902.031931] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 77ca81dc-6322-41de-aaee-adf36d6ce79f] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 902.032457] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 902.032700] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 902.032881] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 902.037675] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 77ca81dc-6322-41de-aaee-adf36d6ce79f] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 902.038353] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 902.038529] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 902.038695] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 902.056747] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 902.056956] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 902.057117] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 907.955099] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] [instance: 9adae455-b609-4ecb-8841-43fb4d826f84] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 4bd4596c-8a2e-47f5-9a3c-6664f1a4b1ac, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 9adae455-b609-4ecb-8841-43fb4d826f84 was re-scheduled: Binding failed for port 4bd4596c-8a2e-47f5-9a3c-6664f1a4b1ac, please check neutron logs for more information.\n'] [ 907.955635] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 907.955883] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9adae455-b609-4ecb-8841-43fb4d826f84.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9adae455-b609-4ecb-8841-43fb4d826f84. [ 907.956551] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] [instance: 9adae455-b609-4ecb-8841-43fb4d826f84] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9adae455-b609-4ecb-8841-43fb4d826f84. [ 907.975933] nova-conductor[53688]: DEBUG nova.network.neutron [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] [instance: 9adae455-b609-4ecb-8841-43fb4d826f84] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 907.990664] nova-conductor[53688]: DEBUG nova.network.neutron [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] [instance: 9adae455-b609-4ecb-8841-43fb4d826f84] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 907.993837] nova-conductor[53688]: DEBUG nova.network.neutron [None req-b068d399-cc70-41c3-a78d-284739a9238a tempest-ServersTestMultiNic-1091074743 tempest-ServersTestMultiNic-1091074743-project-member] [instance: 9adae455-b609-4ecb-8841-43fb4d826f84] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 909.668183] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: 9cedc314-173e-4686-8ee5-28c2512dbcba] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port bfc6fb0b-0208-45a8-ba4f-77ee0a986c82, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 9cedc314-173e-4686-8ee5-28c2512dbcba was re-scheduled: Binding failed for port bfc6fb0b-0208-45a8-ba4f-77ee0a986c82, please check neutron logs for more information.\n'] [ 909.668972] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 909.669669] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9cedc314-173e-4686-8ee5-28c2512dbcba.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9cedc314-173e-4686-8ee5-28c2512dbcba. [ 909.669788] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: 9cedc314-173e-4686-8ee5-28c2512dbcba] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9cedc314-173e-4686-8ee5-28c2512dbcba. [ 909.693443] nova-conductor[53689]: DEBUG nova.network.neutron [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: 9cedc314-173e-4686-8ee5-28c2512dbcba] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 909.723246] nova-conductor[53689]: DEBUG nova.network.neutron [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: 9cedc314-173e-4686-8ee5-28c2512dbcba] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 909.729153] nova-conductor[53689]: DEBUG nova.network.neutron [None req-6aa7edfa-ebcc-4f5b-a570-60fdb481dc84 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: 9cedc314-173e-4686-8ee5-28c2512dbcba] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 910.640173] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Expecting reply to msg 063c5dcc06434ee2b861a5e18bc08f78 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 910.776417] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 063c5dcc06434ee2b861a5e18bc08f78 [ 911.143145] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 911.160537] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 911.160769] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 911.160941] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 911.220239] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 911.220477] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 911.220645] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 911.221017] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 911.221200] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 911.221356] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 911.232347] nova-conductor[53688]: DEBUG nova.quota [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Getting quotas for project 63d2fb7de0ad453dbe6891e6974f1b66. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 911.237072] nova-conductor[53688]: DEBUG nova.quota [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Getting quotas for user 8969ac54b88a47028e5784f6575f2d61 and project 63d2fb7de0ad453dbe6891e6974f1b66. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 911.242550] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: d10134a3-6f70-4f00-b810-371e17d2a1ab] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 911.242991] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 911.243196] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 911.243362] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 911.248617] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: d10134a3-6f70-4f00-b810-371e17d2a1ab] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 911.249260] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 911.249462] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 911.249654] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 911.265964] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 911.266169] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 911.266408] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 911.433707] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] [instance: 31ab5ebd-3df1-4e9f-bf53-69d47176da01] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 228ecc96-89ef-4a4e-8571-06b5011021b9, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 31ab5ebd-3df1-4e9f-bf53-69d47176da01 was re-scheduled: Binding failed for port 228ecc96-89ef-4a4e-8571-06b5011021b9, please check neutron logs for more information.\n'] [ 911.434330] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 911.434646] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 31ab5ebd-3df1-4e9f-bf53-69d47176da01.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 31ab5ebd-3df1-4e9f-bf53-69d47176da01. [ 911.434981] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] [instance: 31ab5ebd-3df1-4e9f-bf53-69d47176da01] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 31ab5ebd-3df1-4e9f-bf53-69d47176da01. [ 911.456183] nova-conductor[53688]: DEBUG nova.network.neutron [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] [instance: 31ab5ebd-3df1-4e9f-bf53-69d47176da01] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 911.471556] nova-conductor[53688]: DEBUG nova.network.neutron [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] [instance: 31ab5ebd-3df1-4e9f-bf53-69d47176da01] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 911.475416] nova-conductor[53688]: DEBUG nova.network.neutron [None req-39cc91ef-e014-4e03-89cc-20e30a69e8e7 tempest-AttachVolumeShelveTestJSON-1091491161 tempest-AttachVolumeShelveTestJSON-1091491161-project-member] [instance: 31ab5ebd-3df1-4e9f-bf53-69d47176da01] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 912.018003] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell0:SELECT=2 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 912.041059] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=4,nova_cell1:INSERT=7 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 912.052297] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=9,nova_api:UPDATE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 912.056114] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=12,nova_api:DELETE=2 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 913.126480] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] [instance: 7dc5b4e9-15c3-4f47-9587-6d34bd44a5ef] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 7a55ee69-0884-433c-9e35-dd184ba21c20, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 7dc5b4e9-15c3-4f47-9587-6d34bd44a5ef was re-scheduled: Binding failed for port 7a55ee69-0884-433c-9e35-dd184ba21c20, please check neutron logs for more information.\n'] [ 913.127014] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 913.127230] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 7dc5b4e9-15c3-4f47-9587-6d34bd44a5ef.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 7dc5b4e9-15c3-4f47-9587-6d34bd44a5ef. [ 913.128110] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] [instance: 7dc5b4e9-15c3-4f47-9587-6d34bd44a5ef] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 7dc5b4e9-15c3-4f47-9587-6d34bd44a5ef. [ 913.162421] nova-conductor[53688]: DEBUG nova.network.neutron [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] [instance: 7dc5b4e9-15c3-4f47-9587-6d34bd44a5ef] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 913.180650] nova-conductor[53688]: DEBUG nova.network.neutron [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] [instance: 7dc5b4e9-15c3-4f47-9587-6d34bd44a5ef] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 913.183952] nova-conductor[53688]: DEBUG nova.network.neutron [None req-3fca92b9-ee3c-4d16-8e90-ac9afc5fe25e tempest-ServerPasswordTestJSON-851135744 tempest-ServerPasswordTestJSON-851135744-project-member] [instance: 7dc5b4e9-15c3-4f47-9587-6d34bd44a5ef] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 913.744834] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Expecting reply to msg 96b121e76ca6477fb4ca4930ff103fd9 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 913.899036] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 96b121e76ca6477fb4ca4930ff103fd9 [ 914.247605] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 914.260714] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 914.260947] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 914.261115] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 914.292238] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 914.292509] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 914.292708] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 914.293115] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 914.293332] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 914.293502] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 914.304165] nova-conductor[53688]: DEBUG nova.quota [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Getting quotas for project c2ffde0eb7394d029b57814b1bc14803. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 914.308048] nova-conductor[53688]: DEBUG nova.quota [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Getting quotas for user f6c0238a6ec4419f9249472a9454c22e and project c2ffde0eb7394d029b57814b1bc14803. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 914.313112] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] [instance: 2178d195-47f4-47ab-9140-b8f849973434] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 914.313606] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 914.313850] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 914.314054] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 914.320456] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] [instance: 2178d195-47f4-47ab-9140-b8f849973434] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 914.320456] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 914.320619] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 914.320749] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 914.336359] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 914.336639] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 914.336851] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 915.035882] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] [instance: d3dafd33-91f8-481d-8f40-8c2e98a7587d] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port b746e4c7-581b-4e6f-8b47-cc4c45268a37, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance d3dafd33-91f8-481d-8f40-8c2e98a7587d was re-scheduled: Binding failed for port b746e4c7-581b-4e6f-8b47-cc4c45268a37, please check neutron logs for more information.\n'] [ 915.036546] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 915.036793] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d3dafd33-91f8-481d-8f40-8c2e98a7587d.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d3dafd33-91f8-481d-8f40-8c2e98a7587d. [ 915.037228] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] [instance: d3dafd33-91f8-481d-8f40-8c2e98a7587d] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d3dafd33-91f8-481d-8f40-8c2e98a7587d. [ 915.084402] nova-conductor[53688]: DEBUG nova.network.neutron [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] [instance: d3dafd33-91f8-481d-8f40-8c2e98a7587d] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 915.113172] nova-conductor[53688]: DEBUG nova.network.neutron [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] [instance: d3dafd33-91f8-481d-8f40-8c2e98a7587d] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 915.120531] nova-conductor[53688]: DEBUG nova.network.neutron [None req-db212900-bf5a-4081-a3f7-a68418d30d6e tempest-ImagesOneServerNegativeTestJSON-358604868 tempest-ImagesOneServerNegativeTestJSON-358604868-project-member] [instance: d3dafd33-91f8-481d-8f40-8c2e98a7587d] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 921.129170] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] [instance: f9e23014-2fe0-4aab-b03c-8759dc1e5eb0] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 1c1b5e97-af36-4179-8875-68e15da56408, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance f9e23014-2fe0-4aab-b03c-8759dc1e5eb0 was re-scheduled: Binding failed for port 1c1b5e97-af36-4179-8875-68e15da56408, please check neutron logs for more information.\n'] [ 921.130009] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 921.130366] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f9e23014-2fe0-4aab-b03c-8759dc1e5eb0.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f9e23014-2fe0-4aab-b03c-8759dc1e5eb0. [ 921.130702] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] [instance: f9e23014-2fe0-4aab-b03c-8759dc1e5eb0] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f9e23014-2fe0-4aab-b03c-8759dc1e5eb0. [ 921.161367] nova-conductor[53688]: DEBUG nova.network.neutron [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] [instance: f9e23014-2fe0-4aab-b03c-8759dc1e5eb0] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 921.180550] nova-conductor[53688]: DEBUG nova.network.neutron [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] [instance: f9e23014-2fe0-4aab-b03c-8759dc1e5eb0] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 921.184332] nova-conductor[53688]: DEBUG nova.network.neutron [None req-d5021fb1-9d31-48d9-94cb-9fc34639326e tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] [instance: f9e23014-2fe0-4aab-b03c-8759dc1e5eb0] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 922.368721] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] [instance: d4a2025d-c128-45a2-b74c-a7fd2630d615] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 39750689-9293-4d1a-b24b-04470262e9e6, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance d4a2025d-c128-45a2-b74c-a7fd2630d615 was re-scheduled: Binding failed for port 39750689-9293-4d1a-b24b-04470262e9e6, please check neutron logs for more information.\n'] [ 922.369399] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 922.369627] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d4a2025d-c128-45a2-b74c-a7fd2630d615.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d4a2025d-c128-45a2-b74c-a7fd2630d615. [ 922.369835] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] [instance: d4a2025d-c128-45a2-b74c-a7fd2630d615] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d4a2025d-c128-45a2-b74c-a7fd2630d615. [ 922.392882] nova-conductor[53688]: DEBUG nova.network.neutron [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] [instance: d4a2025d-c128-45a2-b74c-a7fd2630d615] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 922.410252] nova-conductor[53688]: DEBUG nova.network.neutron [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] [instance: d4a2025d-c128-45a2-b74c-a7fd2630d615] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 922.414982] nova-conductor[53688]: DEBUG nova.network.neutron [None req-5bfe44ca-20c4-4039-8e97-e4fefc8b27b1 tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] [instance: d4a2025d-c128-45a2-b74c-a7fd2630d615] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 924.094755] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: 1e7c4c50-1d33-4947-80bc-ac3ca2453fd5] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 9ea1dfca-8560-4521-8e33-50b5c0f74903, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 1e7c4c50-1d33-4947-80bc-ac3ca2453fd5 was re-scheduled: Binding failed for port 9ea1dfca-8560-4521-8e33-50b5c0f74903, please check neutron logs for more information.\n'] [ 924.095393] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 924.095637] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1e7c4c50-1d33-4947-80bc-ac3ca2453fd5.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1e7c4c50-1d33-4947-80bc-ac3ca2453fd5. [ 924.095872] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: 1e7c4c50-1d33-4947-80bc-ac3ca2453fd5] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 1e7c4c50-1d33-4947-80bc-ac3ca2453fd5. [ 924.132880] nova-conductor[53688]: DEBUG nova.network.neutron [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: 1e7c4c50-1d33-4947-80bc-ac3ca2453fd5] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 924.183209] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Expecting reply to msg 9524e7ac264e4f1bb95345b02a2a3721 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 924.187271] nova-conductor[53688]: DEBUG nova.network.neutron [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: 1e7c4c50-1d33-4947-80bc-ac3ca2453fd5] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 924.191500] nova-conductor[53688]: DEBUG nova.network.neutron [None req-b27a52f7-e161-421f-916e-111f7420cad0 tempest-ImagesTestJSON-1894548440 tempest-ImagesTestJSON-1894548440-project-member] [instance: 1e7c4c50-1d33-4947-80bc-ac3ca2453fd5] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 924.298668] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell0:SELECT=2 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 924.317840] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 9524e7ac264e4f1bb95345b02a2a3721 [ 924.323518] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SELECT=6,nova_cell1:INSERT=15,nova_cell1:SAVEPOINT=1,nova_cell1:RELEASE=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 924.335312] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=9,nova_api:UPDATE=1,nova_api:DELETE=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 924.460608] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Expecting reply to msg 0e42728da7ab4eb0aa3956c596067881 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 924.601158] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 0e42728da7ab4eb0aa3956c596067881 [ 924.687658] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 924.700230] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 924.700462] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 924.700631] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 924.729465] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 924.729716] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 924.729909] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 924.730347] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 924.730749] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 924.730749] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 924.744135] nova-conductor[53688]: DEBUG nova.quota [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Getting quotas for project 4a752424d76840dabab55a9202e7a635. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 924.748362] nova-conductor[53688]: DEBUG nova.quota [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Getting quotas for user 77bd4b765f214cb8bb602e8e52071531 and project 4a752424d76840dabab55a9202e7a635. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 924.757553] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] [instance: 80491222-910b-48e1-a431-3116c336a9a5] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 924.758044] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 924.758261] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 924.758427] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 924.762609] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] [instance: 80491222-910b-48e1-a431-3116c336a9a5] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 924.764160] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 924.764788] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 924.765114] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 924.787613] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 924.787841] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 924.788017] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 924.962670] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 924.978845] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 924.979076] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 924.979240] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 925.029640] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 925.029882] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 925.030263] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 925.030805] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 925.030997] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 925.031573] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 925.042102] nova-conductor[53689]: DEBUG nova.quota [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Getting quotas for project 6b8d7d7387e44003b6b4bc488c7900f3. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 925.045498] nova-conductor[53689]: DEBUG nova.quota [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Getting quotas for user afd2b293ac5747749b0bae2b787277ca and project 6b8d7d7387e44003b6b4bc488c7900f3. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 925.051252] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] [instance: ef2a58d8-74ab-4049-94c3-0eb0d5bfee5f] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 925.051728] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 925.051949] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 925.053344] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 925.057842] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] [instance: ef2a58d8-74ab-4049-94c3-0eb0d5bfee5f] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 925.058515] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 925.058750] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 925.058971] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 925.078392] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 925.078608] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 925.079237] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 931.315187] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SELECT=372,nova_cell1:SAVEPOINT=43,nova_cell1:RELEASE=43,nova_cell1:UPDATE=181,nova_cell1:INSERT=29 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 931.334189] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=370,nova_cell1:UPDATE=179,nova_cell1:INSERT=21,nova_cell1:SAVEPOINT=38,nova_cell1:RELEASE=38 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 932.853271] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: a70f220a-fa34-44af-939f-29292b556897] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 25c7c234-2cad-4819-a66e-0c2b3bc4be20, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance a70f220a-fa34-44af-939f-29292b556897 was re-scheduled: Binding failed for port 25c7c234-2cad-4819-a66e-0c2b3bc4be20, please check neutron logs for more information.\n'] [ 932.854811] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 932.854811] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance a70f220a-fa34-44af-939f-29292b556897.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance a70f220a-fa34-44af-939f-29292b556897. [ 932.854811] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: a70f220a-fa34-44af-939f-29292b556897] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance a70f220a-fa34-44af-939f-29292b556897. [ 932.875930] nova-conductor[53688]: DEBUG nova.network.neutron [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: a70f220a-fa34-44af-939f-29292b556897] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 932.892530] nova-conductor[53688]: DEBUG nova.network.neutron [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: a70f220a-fa34-44af-939f-29292b556897] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 932.895663] nova-conductor[53688]: DEBUG nova.network.neutron [None req-89f2dafa-1d36-4333-a516-37afe2ea854c tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: a70f220a-fa34-44af-939f-29292b556897] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 933.741922] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Expecting reply to msg f031d2cb54b7420fb0d45ccdea52c894 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 933.909475] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg f031d2cb54b7420fb0d45ccdea52c894 [ 934.244447] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 934.257802] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 934.258058] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 934.258240] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 934.283730] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 934.283946] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 934.284135] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 934.284498] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 934.284680] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 934.284837] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 934.294785] nova-conductor[53689]: DEBUG nova.quota [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Getting quotas for project c06b50a7aaa742afbbd0c6fc56c3d131. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 934.297961] nova-conductor[53689]: DEBUG nova.quota [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Getting quotas for user 3078a2af81b248f8b100f58ee66a5a86 and project c06b50a7aaa742afbbd0c6fc56c3d131. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 934.302900] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 4196c9a3-53d2-4a6d-b944-813be319cbdd] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 934.303341] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 934.303533] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 934.303693] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 934.310753] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 4196c9a3-53d2-4a6d-b944-813be319cbdd] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 934.310753] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 934.310929] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 934.312067] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 934.324648] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 934.324793] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 934.324956] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 934.537707] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] [instance: c9214700-faf8-4a26-8084-ffe4a2c06480] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 1d5b23b5-829f-42fc-bf5c-915565c6b728, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance c9214700-faf8-4a26-8084-ffe4a2c06480 was re-scheduled: Binding failed for port 1d5b23b5-829f-42fc-bf5c-915565c6b728, please check neutron logs for more information.\n'] [ 934.538671] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 934.538671] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c9214700-faf8-4a26-8084-ffe4a2c06480.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c9214700-faf8-4a26-8084-ffe4a2c06480. [ 934.538823] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] [instance: c9214700-faf8-4a26-8084-ffe4a2c06480] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c9214700-faf8-4a26-8084-ffe4a2c06480. [ 934.570836] nova-conductor[53689]: DEBUG nova.network.neutron [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] [instance: c9214700-faf8-4a26-8084-ffe4a2c06480] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 934.589191] nova-conductor[53689]: DEBUG nova.network.neutron [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] [instance: c9214700-faf8-4a26-8084-ffe4a2c06480] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 934.593629] nova-conductor[53689]: DEBUG nova.network.neutron [None req-1b0eb2c8-b5c5-4fd1-a35a-82ced94c8ecc tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] [instance: c9214700-faf8-4a26-8084-ffe4a2c06480] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 934.734497] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell0:SELECT=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 934.767724] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SELECT=3,nova_cell1:INSERT=7 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 934.788971] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=6 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 934.789879] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=21,nova_api:DELETE=2,nova_api:UPDATE=2 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 937.312089] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Expecting reply to msg b05c0ebb7dc4483e9cb7c1ac739dfa8c in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 937.446690] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg b05c0ebb7dc4483e9cb7c1ac739dfa8c [ 937.816370] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 937.834526] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 937.834837] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 937.835058] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 937.892918] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 937.893228] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 937.893443] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 937.893862] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 937.894113] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 937.894315] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 937.903512] nova-conductor[53688]: DEBUG nova.quota [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Getting quotas for project a36723dc732b444e8831b049e9f804b8. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 937.906618] nova-conductor[53688]: DEBUG nova.quota [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Getting quotas for user 5345fbbca90446719473829e2ea02386 and project a36723dc732b444e8831b049e9f804b8. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 937.911669] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] [instance: 57a60fa5-be59-48b8-b72a-6f7d945ba821] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 937.912196] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 937.912453] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 937.912665] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 937.921368] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] [instance: 57a60fa5-be59-48b8-b72a-6f7d945ba821] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 937.921368] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 937.921368] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 937.921368] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 937.954590] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 937.954848] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 937.955029] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 940.888256] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 77ca81dc-6322-41de-aaee-adf36d6ce79f] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 7e195214-958a-4642-8567-f18b2b01dc9e, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 77ca81dc-6322-41de-aaee-adf36d6ce79f was re-scheduled: Binding failed for port 7e195214-958a-4642-8567-f18b2b01dc9e, please check neutron logs for more information.\n'] [ 940.888889] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 940.889094] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 77ca81dc-6322-41de-aaee-adf36d6ce79f.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 77ca81dc-6322-41de-aaee-adf36d6ce79f. [ 940.889393] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 77ca81dc-6322-41de-aaee-adf36d6ce79f] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 77ca81dc-6322-41de-aaee-adf36d6ce79f. [ 940.909318] nova-conductor[53688]: DEBUG nova.network.neutron [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 77ca81dc-6322-41de-aaee-adf36d6ce79f] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 940.924140] nova-conductor[53688]: DEBUG nova.network.neutron [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 77ca81dc-6322-41de-aaee-adf36d6ce79f] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 940.926657] nova-conductor[53688]: DEBUG nova.network.neutron [None req-a2ff395f-a2bd-4329-8796-92c04a0ca5ac tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 77ca81dc-6322-41de-aaee-adf36d6ce79f] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 941.885666] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Expecting reply to msg beb6726165ca460e8d87ddd8fafed92e in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 942.006573] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg beb6726165ca460e8d87ddd8fafed92e [ 942.388415] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 942.402076] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 942.402524] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 942.402840] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 942.432504] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 942.432925] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 942.433247] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 942.433744] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 942.434113] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 942.434423] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 942.444976] nova-conductor[53689]: DEBUG nova.quota [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Getting quotas for project d9325f1def284d2a9fdced4e9eeb17f0. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 942.449673] nova-conductor[53689]: DEBUG nova.quota [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Getting quotas for user af34c4e3d81c4729a9dd4a8531992ff1 and project d9325f1def284d2a9fdced4e9eeb17f0. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 942.458699] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: a007a7bc-b898-472b-9469-e6dd6262ea0b] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 942.459777] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 942.460425] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 942.460847] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 942.467417] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: a007a7bc-b898-472b-9469-e6dd6262ea0b] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 942.468388] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 942.468388] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 942.468486] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 942.496131] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 942.496366] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 942.496566] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 943.006858] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: d10134a3-6f70-4f00-b810-371e17d2a1ab] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 6dbb762f-c02f-40ac-8377-f96ca090d475, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance d10134a3-6f70-4f00-b810-371e17d2a1ab was re-scheduled: Binding failed for port 6dbb762f-c02f-40ac-8377-f96ca090d475, please check neutron logs for more information.\n'] [ 943.007472] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 943.007702] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d10134a3-6f70-4f00-b810-371e17d2a1ab.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d10134a3-6f70-4f00-b810-371e17d2a1ab. [ 943.008015] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: d10134a3-6f70-4f00-b810-371e17d2a1ab] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance d10134a3-6f70-4f00-b810-371e17d2a1ab. [ 943.045623] nova-conductor[53689]: DEBUG nova.network.neutron [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: d10134a3-6f70-4f00-b810-371e17d2a1ab] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 943.061397] nova-conductor[53689]: DEBUG nova.network.neutron [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: d10134a3-6f70-4f00-b810-371e17d2a1ab] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 943.063814] nova-conductor[53689]: DEBUG nova.network.neutron [None req-5f94bb74-5383-4919-98dc-8ce9ab823825 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: d10134a3-6f70-4f00-b810-371e17d2a1ab] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 944.553198] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Expecting reply to msg 7adc360177a4422986639a7a1a045108 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 944.698225] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 7adc360177a4422986639a7a1a045108 [ 944.721273] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] [instance: 2178d195-47f4-47ab-9140-b8f849973434] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port becb8351-4f8e-4d86-870c-e7e66f60b598, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 2178d195-47f4-47ab-9140-b8f849973434 was re-scheduled: Binding failed for port becb8351-4f8e-4d86-870c-e7e66f60b598, please check neutron logs for more information.\n'] [ 944.721273] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 944.721273] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 2178d195-47f4-47ab-9140-b8f849973434.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 2178d195-47f4-47ab-9140-b8f849973434. [ 944.721273] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] [instance: 2178d195-47f4-47ab-9140-b8f849973434] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 2178d195-47f4-47ab-9140-b8f849973434. [ 944.748721] nova-conductor[53689]: DEBUG nova.network.neutron [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] [instance: 2178d195-47f4-47ab-9140-b8f849973434] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 944.768894] nova-conductor[53689]: DEBUG nova.network.neutron [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] [instance: 2178d195-47f4-47ab-9140-b8f849973434] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 944.773454] nova-conductor[53689]: DEBUG nova.network.neutron [None req-37c28f3f-2699-4617-9df4-f00a03d7fcd8 tempest-ServerRescueTestJSONUnderV235-2053171794 tempest-ServerRescueTestJSONUnderV235-2053171794-project-member] [instance: 2178d195-47f4-47ab-9140-b8f849973434] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 945.056229] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 945.070966] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 945.071682] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 945.072214] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 945.099993] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 945.100744] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.001s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 945.101148] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 945.101818] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 945.102151] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 945.102595] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 945.111365] nova-conductor[53688]: DEBUG nova.quota [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Getting quotas for project 63d2fb7de0ad453dbe6891e6974f1b66. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 945.114194] nova-conductor[53688]: DEBUG nova.quota [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Getting quotas for user 8969ac54b88a47028e5784f6575f2d61 and project 63d2fb7de0ad453dbe6891e6974f1b66. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 945.120970] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: c24218f8-6989-4e2f-88b4-f4421b66ec3e] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 945.121662] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 945.122017] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 945.122521] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 945.127184] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: c24218f8-6989-4e2f-88b4-f4421b66ec3e] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 945.127971] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 945.128867] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 945.129158] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 945.143843] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 945.144245] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 945.144559] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 948.603303] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] [instance: 80491222-910b-48e1-a431-3116c336a9a5] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 34c92d02-9079-4f11-a58a-625d44ed4cf1, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 80491222-910b-48e1-a431-3116c336a9a5 was re-scheduled: Binding failed for port 34c92d02-9079-4f11-a58a-625d44ed4cf1, please check neutron logs for more information.\n'] [ 948.603856] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 948.604091] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 80491222-910b-48e1-a431-3116c336a9a5.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 80491222-910b-48e1-a431-3116c336a9a5. [ 948.604301] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] [instance: 80491222-910b-48e1-a431-3116c336a9a5] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 80491222-910b-48e1-a431-3116c336a9a5. [ 948.628553] nova-conductor[53689]: DEBUG nova.network.neutron [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] [instance: 80491222-910b-48e1-a431-3116c336a9a5] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 948.643083] nova-conductor[53689]: DEBUG nova.network.neutron [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] [instance: 80491222-910b-48e1-a431-3116c336a9a5] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 948.646523] nova-conductor[53689]: DEBUG nova.network.neutron [None req-e98086e1-3b53-4393-9b50-74f1a5099f1b tempest-AttachInterfacesTestJSON-669022114 tempest-AttachInterfacesTestJSON-669022114-project-member] [instance: 80491222-910b-48e1-a431-3116c336a9a5] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 952.395443] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] [instance: ef2a58d8-74ab-4049-94c3-0eb0d5bfee5f] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port cbe637dd-c959-40d2-af09-5b04853ed211, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance ef2a58d8-74ab-4049-94c3-0eb0d5bfee5f was re-scheduled: Binding failed for port cbe637dd-c959-40d2-af09-5b04853ed211, please check neutron logs for more information.\n'] [ 952.396014] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 952.396255] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ef2a58d8-74ab-4049-94c3-0eb0d5bfee5f.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ef2a58d8-74ab-4049-94c3-0eb0d5bfee5f. [ 952.396464] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] [instance: ef2a58d8-74ab-4049-94c3-0eb0d5bfee5f] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ef2a58d8-74ab-4049-94c3-0eb0d5bfee5f. [ 952.429188] nova-conductor[53689]: DEBUG nova.network.neutron [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] [instance: ef2a58d8-74ab-4049-94c3-0eb0d5bfee5f] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 952.437799] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell0:SELECT=3 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 952.461974] nova-conductor[53689]: DEBUG nova.network.neutron [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] [instance: ef2a58d8-74ab-4049-94c3-0eb0d5bfee5f] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 952.465656] nova-conductor[53689]: DEBUG nova.network.neutron [None req-60fc7659-482e-4c84-81ea-6c3b6fd0ca5e tempest-AttachVolumeNegativeTest-1501631824 tempest-AttachVolumeNegativeTest-1501631824-project-member] [instance: ef2a58d8-74ab-4049-94c3-0eb0d5bfee5f] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 952.475090] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=9,nova_cell1:INSERT=21 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 952.493314] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=19,nova_api:UPDATE=2,nova_api:DELETE=2 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 952.493796] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=17,nova_api:UPDATE=1,nova_api:DELETE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 953.970186] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Expecting reply to msg ebb1f90b36ad4d1ca908657ff7b835c3 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 954.096466] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg ebb1f90b36ad4d1ca908657ff7b835c3 [ 954.473691] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 954.491078] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 954.491344] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 954.491520] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 954.543046] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 954.543274] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 954.543442] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 954.543805] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 954.543989] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 954.544215] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 954.551374] nova-conductor[53688]: DEBUG nova.quota [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Getting quotas for project c4985026e03c486597c6f12f54dfdd56. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 954.553395] nova-conductor[53688]: DEBUG nova.quota [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Getting quotas for user 4551ac61407a4b5297d301e11c44f5fc and project c4985026e03c486597c6f12f54dfdd56. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 954.558124] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] [instance: c181e857-2fcf-4658-9e90-fff396a5eaf5] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 954.558569] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 954.558763] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 954.558925] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 954.564568] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] [instance: c181e857-2fcf-4658-9e90-fff396a5eaf5] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 954.565200] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 954.565452] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 954.565550] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 954.583151] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 954.583415] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 954.583547] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 962.139778] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 4196c9a3-53d2-4a6d-b944-813be319cbdd] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port c5cb1417-be25-47df-b3cd-5ad25361a83b, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 4196c9a3-53d2-4a6d-b944-813be319cbdd was re-scheduled: Binding failed for port c5cb1417-be25-47df-b3cd-5ad25361a83b, please check neutron logs for more information.\n'] [ 962.140492] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 962.140741] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 4196c9a3-53d2-4a6d-b944-813be319cbdd.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 4196c9a3-53d2-4a6d-b944-813be319cbdd. [ 962.141080] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 4196c9a3-53d2-4a6d-b944-813be319cbdd] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 4196c9a3-53d2-4a6d-b944-813be319cbdd. [ 962.167002] nova-conductor[53688]: DEBUG nova.network.neutron [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 4196c9a3-53d2-4a6d-b944-813be319cbdd] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 962.183755] nova-conductor[53688]: DEBUG nova.network.neutron [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 4196c9a3-53d2-4a6d-b944-813be319cbdd] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 962.186433] nova-conductor[53688]: DEBUG nova.network.neutron [None req-19cb45ab-3f47-4269-9973-b7c485bfaebc tempest-DeleteServersTestJSON-485078453 tempest-DeleteServersTestJSON-485078453-project-member] [instance: 4196c9a3-53d2-4a6d-b944-813be319cbdd] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 963.810148] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] [instance: 57a60fa5-be59-48b8-b72a-6f7d945ba821] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port bc0d1c65-eaa1-4791-8213-aad49456dbfb, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 57a60fa5-be59-48b8-b72a-6f7d945ba821 was re-scheduled: Binding failed for port bc0d1c65-eaa1-4791-8213-aad49456dbfb, please check neutron logs for more information.\n'] [ 963.810799] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 963.811066] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 57a60fa5-be59-48b8-b72a-6f7d945ba821.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 57a60fa5-be59-48b8-b72a-6f7d945ba821. [ 963.811325] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] [instance: 57a60fa5-be59-48b8-b72a-6f7d945ba821] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 57a60fa5-be59-48b8-b72a-6f7d945ba821. [ 963.840426] nova-conductor[53689]: DEBUG nova.network.neutron [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] [instance: 57a60fa5-be59-48b8-b72a-6f7d945ba821] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 963.879586] nova-conductor[53689]: DEBUG nova.network.neutron [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] [instance: 57a60fa5-be59-48b8-b72a-6f7d945ba821] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 963.883414] nova-conductor[53689]: DEBUG nova.network.neutron [None req-9fcb21cd-ab67-433a-b992-036f760e3431 tempest-AttachVolumeTestJSON-1728706733 tempest-AttachVolumeTestJSON-1728706733-project-member] [instance: 57a60fa5-be59-48b8-b72a-6f7d945ba821] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 964.547418] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell0:SELECT=3 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 964.567393] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SELECT=9,nova_cell1:INSERT=22,nova_cell1:SAVEPOINT=1,nova_cell1:RELEASE=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 964.578239] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=16,nova_api:UPDATE=1,nova_api:DELETE=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 964.585164] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=20,nova_api:UPDATE=2,nova_api:DELETE=2 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 965.306329] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Expecting reply to msg b2ecc5b430a34897a987397b9b940f20 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 965.441873] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg b2ecc5b430a34897a987397b9b940f20 [ 965.808912] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 965.822116] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 965.822334] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 965.822499] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 965.859898] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 965.860159] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 965.860328] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 965.860687] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 965.860870] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 965.861032] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 965.870911] nova-conductor[53689]: DEBUG nova.quota [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Getting quotas for project fbfacf2c07f543ba85dd1b13ef38f78f. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 965.875170] nova-conductor[53689]: DEBUG nova.quota [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Getting quotas for user 9497e3029e264278bb5f4aac41d83b17 and project fbfacf2c07f543ba85dd1b13ef38f78f. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 965.883455] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] [instance: 518bde7d-d2b2-4b53-b30c-37a7c9d29064] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 965.884363] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 965.884601] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 965.884866] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 965.893806] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] [instance: 518bde7d-d2b2-4b53-b30c-37a7c9d29064] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 965.894684] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 965.894907] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 965.895076] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 965.919126] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 965.919344] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 965.919713] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-84849d30-f59c-4b80-842e-01037ba3cfd2 tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 967.682898] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: a007a7bc-b898-472b-9469-e6dd6262ea0b] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 34ca79ff-567b-43f8-bc85-994e4b2ce95a, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance a007a7bc-b898-472b-9469-e6dd6262ea0b was re-scheduled: Binding failed for port 34ca79ff-567b-43f8-bc85-994e4b2ce95a, please check neutron logs for more information.\n'] [ 967.683501] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 967.683698] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance a007a7bc-b898-472b-9469-e6dd6262ea0b.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance a007a7bc-b898-472b-9469-e6dd6262ea0b. [ 967.683943] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: a007a7bc-b898-472b-9469-e6dd6262ea0b] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance a007a7bc-b898-472b-9469-e6dd6262ea0b. [ 967.703365] nova-conductor[53688]: DEBUG nova.network.neutron [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: a007a7bc-b898-472b-9469-e6dd6262ea0b] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 967.719773] nova-conductor[53688]: DEBUG nova.network.neutron [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: a007a7bc-b898-472b-9469-e6dd6262ea0b] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 967.722567] nova-conductor[53688]: DEBUG nova.network.neutron [None req-eb3e9d2b-3e41-4d16-9930-446e3b8c64e5 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: a007a7bc-b898-472b-9469-e6dd6262ea0b] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 969.184256] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Expecting reply to msg 1426979b753b42c39af205f630e27b01 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 969.243860] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: c24218f8-6989-4e2f-88b4-f4421b66ec3e] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 818e2199-9d00-48ea-a71b-1cb7011eae1e, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance c24218f8-6989-4e2f-88b4-f4421b66ec3e was re-scheduled: Binding failed for port 818e2199-9d00-48ea-a71b-1cb7011eae1e, please check neutron logs for more information.\n'] [ 969.244252] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 969.244403] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c24218f8-6989-4e2f-88b4-f4421b66ec3e.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c24218f8-6989-4e2f-88b4-f4421b66ec3e. [ 969.244612] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: c24218f8-6989-4e2f-88b4-f4421b66ec3e] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c24218f8-6989-4e2f-88b4-f4421b66ec3e. [ 969.268354] nova-conductor[53688]: DEBUG nova.network.neutron [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: c24218f8-6989-4e2f-88b4-f4421b66ec3e] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 969.285170] nova-conductor[53688]: DEBUG nova.network.neutron [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: c24218f8-6989-4e2f-88b4-f4421b66ec3e] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 969.288424] nova-conductor[53688]: DEBUG nova.network.neutron [None req-be1fecdf-ad65-4d82-8eed-1490986ab7bf tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: c24218f8-6989-4e2f-88b4-f4421b66ec3e] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 969.309025] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 1426979b753b42c39af205f630e27b01 [ 969.687685] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 969.700961] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 969.701284] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 969.701510] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 969.733359] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 969.733583] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 969.733794] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 969.734170] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 969.734420] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 969.734591] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 969.744630] nova-conductor[53689]: DEBUG nova.quota [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Getting quotas for project d9325f1def284d2a9fdced4e9eeb17f0. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 969.747037] nova-conductor[53689]: DEBUG nova.quota [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Getting quotas for user af34c4e3d81c4729a9dd4a8531992ff1 and project d9325f1def284d2a9fdced4e9eeb17f0. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 969.752796] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 8e12a771-3033-419b-932e-131821d6e1fe] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 969.753252] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 969.753445] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 969.753606] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 969.757493] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 8e12a771-3033-419b-932e-131821d6e1fe] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 969.758137] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 969.758329] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 969.758527] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 969.773619] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 969.773820] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 969.773984] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 970.865152] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Expecting reply to msg 2ec6ad98343d4334866c5f0388e91b3b in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 970.994672] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 2ec6ad98343d4334866c5f0388e91b3b [ 971.368070] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 971.380913] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 971.381106] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 971.381274] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 971.410446] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 971.410676] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 971.410860] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 971.411257] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 971.411501] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 971.411747] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 971.420490] nova-conductor[53689]: DEBUG nova.quota [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Getting quotas for project 63d2fb7de0ad453dbe6891e6974f1b66. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 971.423989] nova-conductor[53689]: DEBUG nova.quota [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Getting quotas for user 8969ac54b88a47028e5784f6575f2d61 and project 63d2fb7de0ad453dbe6891e6974f1b66. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 971.429230] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: 008a11da-9a85-47d8-9731-602ae35aff64] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 971.429751] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 971.429970] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 971.430158] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 971.434766] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: 008a11da-9a85-47d8-9731-602ae35aff64] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 971.435383] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 971.435571] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 971.435768] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 971.448454] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 971.448665] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 971.448838] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 973.914120] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] [instance: c181e857-2fcf-4658-9e90-fff396a5eaf5] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 124, in wait\n result = hub.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 0b50a699-916e-4d70-a670-340d103c5ef6, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance c181e857-2fcf-4658-9e90-fff396a5eaf5 was re-scheduled: Binding failed for port 0b50a699-916e-4d70-a670-340d103c5ef6, please check neutron logs for more information.\n'] [ 973.914749] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 973.914986] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c181e857-2fcf-4658-9e90-fff396a5eaf5.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c181e857-2fcf-4658-9e90-fff396a5eaf5. [ 973.915221] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] [instance: c181e857-2fcf-4658-9e90-fff396a5eaf5] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c181e857-2fcf-4658-9e90-fff396a5eaf5. [ 973.944858] nova-conductor[53688]: DEBUG nova.network.neutron [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] [instance: c181e857-2fcf-4658-9e90-fff396a5eaf5] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 973.963982] nova-conductor[53688]: DEBUG nova.network.neutron [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] [instance: c181e857-2fcf-4658-9e90-fff396a5eaf5] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 973.972995] nova-conductor[53688]: DEBUG nova.network.neutron [None req-7f581d9d-69d7-4504-833a-429b5824e0b6 tempest-ServersTestFqdnHostnames-142808166 tempest-ServersTestFqdnHostnames-142808166-project-member] [instance: c181e857-2fcf-4658-9e90-fff396a5eaf5] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 977.385226] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-fd221f61-2dfa-44ae-91e8-a689db29b23d tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Targeting cell 84364d98-e6ce-480f-8cca-682929b0326f(cell1) for conductor method rebuild_instance {{(pid=53688) wrapper /opt/stack/nova/nova/conductor/manager.py:95}} [ 977.385500] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-fd221f61-2dfa-44ae-91e8-a689db29b23d tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 977.385648] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-fd221f61-2dfa-44ae-91e8-a689db29b23d tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 977.385815] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-fd221f61-2dfa-44ae-91e8-a689db29b23d tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 977.393495] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-fd221f61-2dfa-44ae-91e8-a689db29b23d tempest-ServerShowV257Test-949954514 tempest-ServerShowV257Test-949954514-project-member] [instance: 518bde7d-d2b2-4b53-b30c-37a7c9d29064] No migration record for the rebuild/evacuate request. {{(pid=53688) rebuild_instance /opt/stack/nova/nova/conductor/manager.py:1227}} [ 981.416139] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell0:SELECT=3 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 981.426811] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=16,nova_api:UPDATE=2,nova_api:DELETE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 981.437537] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=9,nova_cell1:SAVEPOINT=1,nova_cell1:INSERT=22,nova_cell1:RELEASE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 981.447953] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=20,nova_api:DELETE=2,nova_api:UPDATE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 982.665505] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 8e12a771-3033-419b-932e-131821d6e1fe] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 569cb115-6d92-4aca-968c-f05be2b180c7, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 8e12a771-3033-419b-932e-131821d6e1fe was re-scheduled: Binding failed for port 569cb115-6d92-4aca-968c-f05be2b180c7, please check neutron logs for more information.\n'] [ 982.666247] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 982.666990] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 8e12a771-3033-419b-932e-131821d6e1fe.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 8e12a771-3033-419b-932e-131821d6e1fe. [ 982.667278] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 8e12a771-3033-419b-932e-131821d6e1fe] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 8e12a771-3033-419b-932e-131821d6e1fe. [ 982.686790] nova-conductor[53689]: DEBUG nova.network.neutron [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 8e12a771-3033-419b-932e-131821d6e1fe] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 982.701562] nova-conductor[53689]: DEBUG nova.network.neutron [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 8e12a771-3033-419b-932e-131821d6e1fe] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 982.704420] nova-conductor[53689]: DEBUG nova.network.neutron [None req-70a66f56-f283-4b88-a63c-89f803cca1fe tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 8e12a771-3033-419b-932e-131821d6e1fe] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 983.764458] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Expecting reply to msg bbe27e6b093d4992b284842bfb5a89ec in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 983.891133] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg bbe27e6b093d4992b284842bfb5a89ec [ 984.266971] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Took 0.50 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 984.280664] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 984.280869] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 984.281027] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 984.313766] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 984.314021] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 984.314188] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 984.314613] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 984.314740] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 984.314900] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 984.321971] nova-conductor[53689]: DEBUG nova.quota [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Getting quotas for project d9325f1def284d2a9fdced4e9eeb17f0. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 984.326599] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: 008a11da-9a85-47d8-9731-602ae35aff64] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port fafa6f04-80b2-4750-8cda-117f48969222, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 008a11da-9a85-47d8-9731-602ae35aff64 was re-scheduled: Binding failed for port fafa6f04-80b2-4750-8cda-117f48969222, please check neutron logs for more information.\n'] [ 984.326599] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 984.326866] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 008a11da-9a85-47d8-9731-602ae35aff64.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 008a11da-9a85-47d8-9731-602ae35aff64. [ 984.326969] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: 008a11da-9a85-47d8-9731-602ae35aff64] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 008a11da-9a85-47d8-9731-602ae35aff64. [ 984.330782] nova-conductor[53689]: DEBUG nova.quota [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Getting quotas for user af34c4e3d81c4729a9dd4a8531992ff1 and project d9325f1def284d2a9fdced4e9eeb17f0. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 984.346961] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 69e7aa90-75af-46ea-95ca-bca19e36dfec] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 984.347430] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 984.347644] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 984.347811] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 984.354470] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 69e7aa90-75af-46ea-95ca-bca19e36dfec] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 984.355114] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 984.355286] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 984.355447] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 984.357036] nova-conductor[53689]: DEBUG nova.network.neutron [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: 008a11da-9a85-47d8-9731-602ae35aff64] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 984.372558] nova-conductor[53689]: DEBUG nova.network.neutron [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: 008a11da-9a85-47d8-9731-602ae35aff64] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 984.377431] nova-conductor[53689]: DEBUG nova.network.neutron [None req-fc1558ec-d221-4e7e-918f-d923a306b490 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: 008a11da-9a85-47d8-9731-602ae35aff64] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 984.387726] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 984.387943] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 984.388124] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 985.439024] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Expecting reply to msg 91298df74b6a44128b524e9d1babaa62 in queue reply_d19a0b27ff1248c0876daa8ff85cc326 [ 985.570963] nova-conductor[53688]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 91298df74b6a44128b524e9d1babaa62 [ 985.943056] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53688) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 985.956815] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 985.957050] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 985.957221] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 986.002626] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 986.002626] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 986.002626] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 986.002626] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 986.002626] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 986.002626] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 986.005304] nova-conductor[53688]: DEBUG nova.quota [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Getting quotas for project 63d2fb7de0ad453dbe6891e6974f1b66. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 986.007387] nova-conductor[53688]: DEBUG nova.quota [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Getting quotas for user 8969ac54b88a47028e5784f6575f2d61 and project 63d2fb7de0ad453dbe6891e6974f1b66. Resources: {'cores', 'ram', 'instances'} {{(pid=53688) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 986.012257] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: f1051b8f-fcfc-4aaa-8e63-9fafbb8f5828] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53688) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 986.012720] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 986.012941] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 986.013109] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 986.017306] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: f1051b8f-fcfc-4aaa-8e63-9fafbb8f5828] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53688) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 986.017942] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 986.018135] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 986.018298] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 986.031823] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 986.032088] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 986.032272] nova-conductor[53688]: DEBUG oslo_concurrency.lockutils [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53688) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 991.361138] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SAVEPOINT=34,nova_cell1:RELEASE=34,nova_cell1:UPDATE=193,nova_cell1:SELECT=391,nova_cell1:INSERT=32 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 991.676272] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SAVEPOINT=40,nova_cell1:RELEASE=40,nova_cell1:SELECT=339,nova_cell1:UPDATE=164,nova_cell1:INSERT=20 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 994.318165] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell0:SELECT=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 994.359427] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=3,nova_cell1:INSERT=7 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 994.364443] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=5 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 994.376427] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=7,nova_api:UPDATE=1,nova_api:DELETE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 996.001820] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell0:SELECT=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 996.020773] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:SELECT=7,nova_cell1:UPDATE=3,nova_cell1:INSERT=8 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 996.027399] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=8 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 996.030313] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_api:SELECT=6,nova_api:UPDATE=1,nova_api:DELETE=1 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 997.163026] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 69e7aa90-75af-46ea-95ca-bca19e36dfec] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 43474987-c405-46c0-8e73-9c430b8fa49a, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 69e7aa90-75af-46ea-95ca-bca19e36dfec was re-scheduled: Binding failed for port 43474987-c405-46c0-8e73-9c430b8fa49a, please check neutron logs for more information.\n'] [ 997.164692] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 997.164921] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 69e7aa90-75af-46ea-95ca-bca19e36dfec.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 69e7aa90-75af-46ea-95ca-bca19e36dfec. [ 997.165238] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 69e7aa90-75af-46ea-95ca-bca19e36dfec] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 69e7aa90-75af-46ea-95ca-bca19e36dfec. [ 997.203682] nova-conductor[53688]: DEBUG nova.network.neutron [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 69e7aa90-75af-46ea-95ca-bca19e36dfec] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 997.220797] nova-conductor[53688]: DEBUG nova.network.neutron [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 69e7aa90-75af-46ea-95ca-bca19e36dfec] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 997.224373] nova-conductor[53688]: DEBUG nova.network.neutron [None req-e1075c80-68e3-4692-a113-629c2181b7d8 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 69e7aa90-75af-46ea-95ca-bca19e36dfec] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 998.287691] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Expecting reply to msg 069e1578f16943608a2e1ec5dbfd9e71 in queue reply_5133ae70522a4309b9b1926cdadcc99f [ 998.426356] nova-conductor[53689]: INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 069e1578f16943608a2e1ec5dbfd9e71 [ 998.791528] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Took 0.51 seconds to select destinations for 1 instance(s). {{(pid=53689) _schedule_instances /opt/stack/nova/nova/conductor/manager.py:945}} [ 998.804016] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 998.804252] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 998.804418] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 998.835275] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "00000000-0000-0000-0000-000000000000" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 998.835498] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 998.835679] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "00000000-0000-0000-0000-000000000000" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 998.836075] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 998.836294] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 998.836698] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 998.845838] nova-conductor[53689]: DEBUG nova.quota [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Getting quotas for project d9325f1def284d2a9fdced4e9eeb17f0. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:393}} [ 998.849060] nova-conductor[53689]: DEBUG nova.quota [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Getting quotas for user af34c4e3d81c4729a9dd4a8531992ff1 and project d9325f1def284d2a9fdced4e9eeb17f0. Resources: {'cores', 'ram', 'instances'} {{(pid=53689) _get_quotas /opt/stack/nova/nova/quota.py:383}} [ 998.855272] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 74f79e89-3508-474f-ac4e-cc202231eed7] Selected host: cpu-1; Selected node: domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28; Alternates: [] {{(pid=53689) schedule_and_build_instances /opt/stack/nova/nova/conductor/manager.py:1767}} [ 998.855793] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 998.856059] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 998.856241] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 998.870203] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 74f79e89-3508-474f-ac4e-cc202231eed7] block_device_mapping [BlockDeviceMapping(attachment_id=,boot_index=0,connection_info=None,created_at=,delete_on_termination=True,deleted=,deleted_at=,destination_type='local',device_name=None,device_type='disk',disk_bus=None,encrypted=False,encryption_format=None,encryption_options=None,encryption_secret_uuid=None,guest_format=None,id=,image_id='4a9e718e-a6a1-4b4a-b567-8e55529b2d5b',instance=,instance_uuid=,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=,uuid=,volume_id=None,volume_size=None,volume_type=None)] {{(pid=53689) _create_block_device_mapping /opt/stack/nova/nova/conductor/manager.py:1512}} [ 998.870887] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 998.871083] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 998.871356] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 998.900724] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Acquiring lock "84364d98-e6ce-480f-8cca-682929b0326f" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:402}} [ 998.900947] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" acquired by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:407}} [ 998.901118] nova-conductor[53689]: DEBUG oslo_concurrency.lockutils [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Lock "84364d98-e6ce-480f-8cca-682929b0326f" "released" by "nova.context.set_target_cell..get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=53689) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:421}} [ 999.755343] nova-conductor[53688]: ERROR nova.scheduler.utils [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: f1051b8f-fcfc-4aaa-8e63-9fafbb8f5828] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 1d1878be-2615-4a0e-9c52-5ae375ffee0f, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance f1051b8f-fcfc-4aaa-8e63-9fafbb8f5828 was re-scheduled: Binding failed for port 1d1878be-2615-4a0e-9c52-5ae375ffee0f, please check neutron logs for more information.\n'] [ 999.755861] nova-conductor[53688]: DEBUG nova.conductor.manager [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Rescheduling: True {{(pid=53688) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 999.756110] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f1051b8f-fcfc-4aaa-8e63-9fafbb8f5828.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f1051b8f-fcfc-4aaa-8e63-9fafbb8f5828. [ 999.756327] nova-conductor[53688]: WARNING nova.scheduler.utils [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: f1051b8f-fcfc-4aaa-8e63-9fafbb8f5828] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f1051b8f-fcfc-4aaa-8e63-9fafbb8f5828. [ 999.778564] nova-conductor[53688]: DEBUG nova.network.neutron [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: f1051b8f-fcfc-4aaa-8e63-9fafbb8f5828] deallocate_for_instance() {{(pid=53688) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 999.794474] nova-conductor[53688]: DEBUG nova.network.neutron [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: f1051b8f-fcfc-4aaa-8e63-9fafbb8f5828] Instance cache missing network info. {{(pid=53688) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 999.798329] nova-conductor[53688]: DEBUG nova.network.neutron [None req-3f15f766-fb75-462f-b6c8-fd76785b1b98 tempest-ServerDiskConfigTestJSON-2141274115 tempest-ServerDiskConfigTestJSON-2141274115-project-member] [instance: f1051b8f-fcfc-4aaa-8e63-9fafbb8f5828] Updating instance_info_cache with network_info: [] {{(pid=53688) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 1008.841324] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell0:SELECT=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 1008.874415] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SELECT=3,nova_cell1:INSERT=7 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 1008.894423] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=7,nova_api:UPDATE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 1008.899233] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_api:SELECT=5,nova_api:DELETE=1 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 1011.364726] nova-conductor[53689]: ERROR nova.scheduler.utils [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 74f79e89-3508-474f-ac4e-cc202231eed7] Error from last host: cpu-1 (node domain-c8.fc996f14-c53b-4953-92e3-bdfa48f5cc28): ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2641, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 539, in spawn\n self._vmops.spawn(context, instance, image_meta, injected_files,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn\n vm_ref = self.build_virtual_machine(instance,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 275, in build_virtual_machine\n vif_infos = vmwarevif.get_vif_info(self._session,\n', ' File "/opt/stack/nova/nova/virt/vmwareapi/vif.py", line 119, in get_vif_info\n for vif in network_info:\n', ' File "/opt/stack/nova/nova/network/model.py", line 612, in __iter__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/model.py", line 603, in _sync_wrapper\n self.wait()\n', ' File "/opt/stack/nova/nova/network/model.py", line 635, in wait\n self[:] = self._gt.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 225, in wait\n return self._exit_event.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 131, in wait\n current.throw(*self._exc)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main\n result = function(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2011, in _allocate_network_async\n raise e\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 1989, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1229, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1365, in _update_ports_for_instance\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 1340, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 585, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/opt/stack/nova/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port 4a2145c6-50f5-4f1c-86d8-12ac61bec027, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2456, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 2748, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 74f79e89-3508-474f-ac4e-cc202231eed7 was re-scheduled: Binding failed for port 4a2145c6-50f5-4f1c-86d8-12ac61bec027, please check neutron logs for more information.\n'] [ 1011.365490] nova-conductor[53689]: DEBUG nova.conductor.manager [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Rescheduling: True {{(pid=53689) build_instances /opt/stack/nova/nova/conductor/manager.py:695}} [ 1011.365782] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 74f79e89-3508-474f-ac4e-cc202231eed7.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 74f79e89-3508-474f-ac4e-cc202231eed7. [ 1011.366152] nova-conductor[53689]: WARNING nova.scheduler.utils [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 74f79e89-3508-474f-ac4e-cc202231eed7] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 74f79e89-3508-474f-ac4e-cc202231eed7. [ 1011.400655] nova-conductor[53689]: DEBUG nova.network.neutron [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 74f79e89-3508-474f-ac4e-cc202231eed7] deallocate_for_instance() {{(pid=53689) deallocate_for_instance /opt/stack/nova/nova/network/neutron.py:1803}} [ 1011.415781] nova-conductor[53689]: DEBUG nova.network.neutron [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 74f79e89-3508-474f-ac4e-cc202231eed7] Instance cache missing network info. {{(pid=53689) _get_preexisting_port_ids /opt/stack/nova/nova/network/neutron.py:3323}} [ 1011.421252] nova-conductor[53689]: DEBUG nova.network.neutron [None req-bdedfe07-2453-4e3d-8f01-5d69cf9903a4 tempest-ServersTestJSON-462811407 tempest-ServersTestJSON-462811407-project-member] [instance: 74f79e89-3508-474f-ac4e-cc202231eed7] Updating instance_info_cache with network_info: [] {{(pid=53689) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:116}} [ 1021.383606] nova-conductor[53688]: DEBUG dbcounter [-] [53688] Writing DB stats nova_cell1:UPDATE=46,nova_cell1:SELECT=73,nova_cell1:DELETE=2,nova_cell1:SAVEPOINT=8,nova_cell1:RELEASE=8,nova_cell1:INSERT=4 {{(pid=53688) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}} [ 1021.430657] nova-conductor[53689]: DEBUG dbcounter [-] [53689] Writing DB stats nova_cell1:SAVEPOINT=8,nova_cell1:RELEASE=8,nova_cell1:SELECT=48,nova_cell1:UPDATE=26,nova_cell1:INSERT=4 {{(pid=53689) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:115}}