[ 10.420017][ C0] clocksource: Watchdog remote CPU 3 read timed out [ 10.420105][ C0] [ 10.420107][ C0] ======================================================== [ 10.420108][ C0] WARNING: possible irq lock inversion dependency detected [ 10.420110][ C0] 7.2.0-virtme #1 Tainted: G W [ 10.420112][ C0] -------------------------------------------------------- [ 10.420112][ C0] swapper/0/0 just changed the state of lock: [ 10.420113][ C0] ffffffff8b489de0 (console_owner){..-.}-{0:0}, at: console_lock_spinning_enable+0x40/0x60 [ 10.420125][ C0] but this lock took another, SOFTIRQ-unsafe lock in the past: [ 10.420126][ C0] (fs_reclaim){+.+.}-{0:0} [ 10.420128][ C0] [ 10.420128][ C0] [ 10.420128][ C0] and interrupts could create inverse lock ordering between them. [ 10.420128][ C0] [ 10.420128][ C0] [ 10.420128][ C0] other info that might help us debug this: [ 10.420129][ C0] Possible interrupt unsafe locking scenario: [ 10.420129][ C0] [ 10.420130][ C0] CPU0 CPU1 [ 10.420130][ C0] ---- ---- [ 10.420130][ C0] lock(fs_reclaim); [ 10.420131][ C0] local_irq_disable(); [ 10.420132][ C0] lock(console_owner); [ 10.420133][ C0] lock(fs_reclaim); [ 10.420133][ C0] [ 10.420134][ C0] lock(console_owner); [ 10.420135][ C0] [ 10.420135][ C0] *** DEADLOCK *** [ 10.420135][ C0] [ 10.420135][ C0] locks held by swapper/0/0: 4, last CPU#0: [ 10.420136][ C0] #0: ffa0000000007c90 ((&watchdog_timer)){+.-.}-{0:0}, at: call_timer_fn+0x110/0x4d0 [ 10.420141][ C0] #1: ffffffff8b5cf0b8 (watchdog_lock){+.-.}-{3:3}, at: clocksource_watchdog+0x15/0x30 [ 10.420145][ C0] #2: ffffffff8b56a440 (console_lock){+.+.}-{0:0}, at: vprintk_emit+0x320/0x3e0 [ 10.420148][ C0] #3: ffffffff8b56a4b8 (console_srcu){....}-{0:0}, at: console_flush_one_record+0x106/0x710 [ 10.420151][ C0] [ 10.420151][ C0] the shortest dependencies between 2nd lock and 1st lock: [ 10.420156][ C0] -> (fs_reclaim){+.+.}-{0:0} { [ 10.420158][ C0] HARDIRQ-ON-W at: [ 10.420159][ C0] __lock_acquire+0x388/0xc20 [ 10.420163][ C0] lock_acquire.part.0+0xd4/0x280 [ 10.420164][ C0] fs_reclaim_acquire+0xd5/0x120 [ 10.420167][ C0] __kmalloc_cache_noprof+0x6e/0x620 [ 10.420170][ C0] kthread_create_worker_on_node+0xea/0x210 [ 10.420173][ C0] workqueue_init+0x2a/0x680 [ 10.420176][ C0] kernel_init_freeable+0x2fe/0x630 [ 10.420179][ C0] kernel_init+0x21/0x150 [ 10.420182][ C0] ret_from_fork+0x4a4/0x720 [ 10.420186][ C0] ret_from_fork_asm+0x11/0x20 [ 10.420189][ C0] SOFTIRQ-ON-W at: [ 10.420189][ C0] __lock_acquire+0x388/0xc20 [ 10.420191][ C0] lock_acquire.part.0+0xd4/0x280 [ 10.420193][ C0] fs_reclaim_acquire+0xd5/0x120 [ 10.420194][ C0] __kmalloc_cache_noprof+0x6e/0x620 [ 10.420195][ C0] kthread_create_worker_on_node+0xea/0x210 [ 10.420197][ C0] workqueue_init+0x2a/0x680 [ 10.420199][ C0] kernel_init_freeable+0x2fe/0x630 [ 10.420200][ C0] kernel_init+0x21/0x150 [ 10.420201][ C0] ret_from_fork+0x4a4/0x720 [ 10.420202][ C0] ret_from_fork_asm+0x11/0x20 [ 10.420204][ C0] INITIAL USE at: [ 10.420204][ C0] __lock_acquire+0x388/0xc20 [ 10.420206][ C0] lock_acquire.part.0+0xd4/0x280 [ 10.420208][ C0] fs_reclaim_acquire+0xd5/0x120 [ 10.420209][ C0] __kmalloc_cache_noprof+0x6e/0x620 [ 10.420210][ C0] kthread_create_worker_on_node+0xea/0x210 [ 10.420212][ C0] workqueue_init+0x2a/0x680 [ 10.420213][ C0] kernel_init_freeable+0x2fe/0x630 [ 10.420215][ C0] kernel_init+0x21/0x150 [ 10.420216][ C0] ret_from_fork+0x4a4/0x720 [ 10.420217][ C0] ret_from_fork_asm+0x11/0x20 [ 10.420219][ C0] } [ 10.420219][ C0] ... key at: [] __fs_reclaim_map+0x0/0xbe0 [ 10.420223][ C0] ... acquired at: [ 10.420224][ C0] __lock_acquire+0x518/0xc20 [ 10.420226][ C0] lock_acquire.part.0+0xd4/0x280 [ 10.420227][ C0] fs_reclaim_acquire+0xd5/0x120 [ 10.420228][ C0] __kmalloc_noprof+0xd3/0x760 [ 10.420230][ C0] alloc_buf.isra.0+0x4b/0x260 [ 10.420233][ C0] put_chars+0x1e1/0x2f0 [ 10.420235][ C0] hvc_console_print+0x292/0x780 [ 10.420237][ C0] console_emit_next_record+0x22f/0x4c0 [ 10.420238][ C0] console_flush_one_record+0x46f/0x710 [ 10.420239][ C0] console_unlock+0xee/0x1f0 [ 10.420240][ C0] vprintk_emit+0x37c/0x3e0 [ 10.420241][ C0] dev_vprintk_emit+0x27f/0x2c0 [ 10.420244][ C0] dev_printk_emit+0xb9/0xee [ 10.420246][ C0] _dev_info+0xe2/0x116 [ 10.420248][ C0] __devm_rtc_register_device.cold+0xc3/0x3a8 [ 10.420251][ C0] cmos_do_probe+0x73b/0x98a [ 10.420252][ C0] platform_probe+0xfe/0x1f0 [ 10.420254][ C0] call_driver_probe+0x61/0x1c0 [ 10.420257][ C0] really_probe+0x199/0x760 [ 10.420258][ C0] __driver_probe_device+0x24f/0x440 [ 10.420260][ C0] driver_probe_device+0x4a/0xf0 [ 10.420262][ C0] __driver_attach+0x1b8/0x540 [ 10.420263][ C0] bus_for_each_dev+0x130/0x1e0 [ 10.420265][ C0] bus_add_driver+0x2c8/0x530 [ 10.420266][ C0] driver_register+0x1a3/0x390 [ 10.420268][ C0] __platform_driver_probe+0x13f/0x270 [ 10.420269][ C0] cmos_init+0x31/0x40 [ 10.420272][ C0] do_one_initcall+0x124/0x4f0 [ 10.420273][ C0] kernel_init_freeable+0x596/0x630 [ 10.420274][ C0] kernel_init+0x21/0x150 [ 10.420275][ C0] ret_from_fork+0x4a4/0x720 [ 10.420277][ C0] ret_from_fork_asm+0x11/0x20 [ 10.420278][ C0] [ 10.420279][ C0] -> (console_owner){..-.}-{0:0} { [ 10.420281][ C0] IN-SOFTIRQ-W at: [ 10.420281][ C0] __lock_acquire+0x388/0xc20 [ 10.420283][ C0] lock_acquire.part.0+0xd4/0x280 [ 10.420284][ C0] console_lock_spinning_enable+0x5c/0x60 [ 10.420286][ C0] console_emit_next_record+0x1d1/0x4c0 [ 10.420287][ C0] console_flush_one_record+0x46f/0x710 [ 10.420288][ C0] console_unlock+0xee/0x1f0 [ 10.420289][ C0] vprintk_emit+0x37c/0x3e0 [ 10.420290][ C0] _printk+0xc7/0x100 [ 10.420293][ C0] clocksource_watchdog.part.0.cold+0x105/0x1dd [ 10.420295][ C0] call_timer_fn+0x160/0x4d0 [ 10.420296][ C0] __run_timers+0x68f/0xaa0 [ 10.420297][ C0] run_timer_softirq+0xf0/0x160 [ 10.420299][ C0] handle_softirqs+0x1d3/0x900 [ 10.420301][ C0] __irq_exit_rcu+0x145/0x1c0 [ 10.420303][ C0] irq_exit_rcu+0xe/0x30 [ 10.420304][ C0] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 10.420307][ C0] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 10.420309][ C0] pv_native_safe_halt+0xf/0x10 [ 10.420310][ C0] default_idle+0x9/0x10 [ 10.420311][ C0] default_idle_call+0x6e/0xb0 [ 10.420312][ C0] cpuidle_idle_call.constprop.0+0x237/0x410 [ 10.420315][ C0] do_idle+0xd8/0x190 [ 10.420317][ C0] cpu_startup_entry+0x53/0x70 [ 10.420319][ C0] rest_init+0x279/0x280 [ 10.420320][ C0] start_kernel+0x3af/0x3b0 [ 10.420321][ C0] x86_64_start_reservations+0x24/0x30 [ 10.420323][ C0] x86_64_start_kernel+0x12b/0x130 [ 10.420324][ C0] common_startup_64+0x13e/0x148 [ 10.420326][ C0] INITIAL USE at: [ 10.420327][ C0] } [ 10.420328][ C0] ... key at: [] console_owner_dep_map+0x0/0x60 [ 10.420331][ C0] ... acquired at: [ 10.420331][ C0] mark_lock+0x1d7/0xa00 [ 10.420333][ C0] mark_usage+0x42/0x170 [ 10.420334][ C0] __lock_acquire+0x388/0xc20 [ 10.420336][ C0] lock_acquire.part.0+0xd4/0x280 [ 10.420337][ C0] console_lock_spinning_enable+0x5c/0x60 [ 10.420338][ C0] console_emit_next_record+0x1d1/0x4c0 [ 10.420339][ C0] console_flush_one_record+0x46f/0x710 [ 10.420340][ C0] console_unlock+0xee/0x1f0 [ 10.420341][ C0] vprintk_emit+0x37c/0x3e0 [ 10.420342][ C0] _printk+0xc7/0x100 [ 10.420343][ C0] clocksource_watchdog.part.0.cold+0x105/0x1dd [ 10.420344][ C0] call_timer_fn+0x160/0x4d0 [ 10.420345][ C0] __run_timers+0x68f/0xaa0 [ 10.420346][ C0] run_timer_softirq+0xf0/0x160 [ 10.420347][ C0] handle_softirqs+0x1d3/0x900 [ 10.420349][ C0] __irq_exit_rcu+0x145/0x1c0 [ 10.420350][ C0] irq_exit_rcu+0xe/0x30 [ 10.420351][ C0] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 10.420353][ C0] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 10.420354][ C0] pv_native_safe_halt+0xf/0x10 [ 10.420356][ C0] default_idle+0x9/0x10 [ 10.420357][ C0] default_idle_call+0x6e/0xb0 [ 10.420358][ C0] cpuidle_idle_call.constprop.0+0x237/0x410 [ 10.420359][ C0] do_idle+0xd8/0x190 [ 10.420360][ C0] cpu_startup_entry+0x53/0x70 [ 10.420362][ C0] rest_init+0x279/0x280 [ 10.420363][ C0] start_kernel+0x3af/0x3b0 [ 10.420364][ C0] x86_64_start_reservations+0x24/0x30 [ 10.420365][ C0] x86_64_start_kernel+0x12b/0x130 [ 10.420366][ C0] common_startup_64+0x13e/0x148 [ 10.420367][ C0] [ 10.420368][ C0] [ 10.420368][ C0] stack backtrace: [ 10.420370][ C0] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 7.2.0-virtme #1 PREEMPT(full) [ 10.420373][ C0] Tainted: [W]=WARN [ 10.420374][ C0] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 10.420376][ C0] Call Trace: [ 10.420377][ C0] [ 10.420379][ C0] dump_stack_lvl+0x6f/0xa0 [ 10.420382][ C0] print_irq_inversion_bug.part.0.cold+0xe6/0x143 [ 10.420384][ C0] mark_lock_irq+0x989/0x9c0 [ 10.420386][ C0] ? add_lock_to_list+0x2c/0x1a0 [ 10.420389][ C0] mark_lock+0x1d7/0xa00 [ 10.420391][ C0] mark_usage+0x42/0x170 [ 10.420393][ C0] __lock_acquire+0x388/0xc20 [ 10.420396][ C0] lock_acquire.part.0+0xd4/0x280 [ 10.420397][ C0] ? console_lock_spinning_enable+0x40/0x60 [ 10.420399][ C0] ? rcu_is_watching+0x16/0xd0 [ 10.420401][ C0] ? lock_acquire+0x13c/0x160 [ 10.420404][ C0] console_lock_spinning_enable+0x5c/0x60 [ 10.420405][ C0] ? console_lock_spinning_enable+0x40/0x60 [ 10.420406][ C0] console_emit_next_record+0x1d1/0x4c0 [ 10.420408][ C0] ? devkmsg_read+0x4b0/0x4b0 [ 10.420409][ C0] ? console_flush_one_record+0x106/0x710 [ 10.420410][ C0] ? rcu_is_watching+0x16/0xd0 [ 10.420411][ C0] ? lock_acquire+0x13c/0x160 [ 10.420414][ C0] console_flush_one_record+0x46f/0x710 [ 10.420415][ C0] ? console_emit_next_record+0x4c0/0x4c0 [ 10.420416][ C0] ? __lock_acquire+0x518/0xc20 [ 10.420419][ C0] console_unlock+0xee/0x1f0 [ 10.420420][ C0] ? console_flush_one_record+0x710/0x710 [ 10.420421][ C0] ? rcu_is_watching+0x16/0xd0 [ 10.420423][ C0] ? lock_acquire+0x60/0x160 [ 10.420425][ C0] ? __down_trylock_console_sem+0x5e/0xa0 [ 10.420427][ C0] ? vprintk_emit+0x320/0x3e0 [ 10.420428][ C0] vprintk_emit+0x37c/0x3e0 [ 10.420430][ C0] ? wake_up_klogd_work_func+0x90/0x90 [ 10.420432][ C0] ? clocksource_watchdog.part.0+0x450/0x540 [ 10.420434][ C0] _printk+0xc7/0x100 [ 10.420436][ C0] ? snapshot_read.cold+0x21/0x21 [ 10.420437][ C0] ? clocksource_watchdog.part.0+0x450/0x540 [ 10.420439][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 10.420442][ C0] clocksource_watchdog.part.0.cold+0x105/0x1dd [ 10.420443][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 10.420445][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 10.420447][ C0] call_timer_fn+0x160/0x4d0 [ 10.420449][ C0] ? detach_if_pending+0x1d0/0x1d0 [ 10.420450][ C0] ? debug_object_active_state+0x430/0x430 [ 10.420453][ C0] ? find_held_lock+0x2b/0x80 [ 10.420455][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 10.420457][ C0] ? rcu_is_watching+0x16/0xd0 [ 10.420459][ C0] __run_timers+0x68f/0xaa0 [ 10.420460][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 10.420463][ C0] ? __bpf_trace_itimer_expire+0x10/0x10 [ 10.420464][ C0] ? __lock_acquire+0x518/0xc20 [ 10.420467][ C0] ? __rwlock_init+0x150/0x150 [ 10.420470][ C0] run_timer_softirq+0xf0/0x160 [ 10.420471][ C0] ? __run_timers+0xaa0/0xaa0 [ 10.420472][ C0] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 10.420475][ C0] ? rcu_is_watching+0x16/0xd0 [ 10.420476][ C0] handle_softirqs+0x1d3/0x900 [ 10.420478][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 10.420480][ C0] ? _local_bh_enable+0xc0/0xc0 [ 10.420482][ C0] __irq_exit_rcu+0x145/0x1c0 [ 10.420484][ C0] irq_exit_rcu+0xe/0x30 [ 10.420485][ C0] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 10.420487][ C0] [ 10.420488][ C0] [ 10.420488][ C0] ? lockdep_hardirqs_on_prepare.part.0+0x9a/0x160 [ 10.420490][ C0] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 10.420492][ C0] RIP: 0010:pv_native_safe_halt+0xf/0x10 [ 10.420495][ C0] Code: 48 8b 3d d4 41 10 02 e8 1f 00 00 00 48 2b 05 98 72 b2 00 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa eb 07 0f 00 2d 53 45 29 00 fb f4 0f 1f 40 d6 48 83 ec 20 8b 17 49 89 f8 83 e2 fe 41 89 d2 0f 01 [ 10.420496][ C0] RSP: 0018:ffffffff8b207cf8 EFLAGS: 00000296 [ 10.420498][ C0] RAX: 000000000004fbfb RBX: ffffffff8b21c600 RCX: ffffffff880f4ce7 [ 10.420500][ C0] RDX: ffffffff8b21c600 RSI: ffffffff8b088c54 RDI: ffffffff8aa93160 [ 10.420501][ C0] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 10.420501][ C0] R10: 0000000000000000 R11: 0000000000000001 R12: 1ffffffff1640fa2 [ 10.420502][ C0] R13: 0000000000000000 R14: dffffc0000000000 R15: 0000000000014770 [ 10.420504][ C0] ? cpuidle_idle_call.constprop.0+0x237/0x410 [ 10.420506][ C0] ? lockdep_hardirqs_on+0x91/0x130 [ 10.420508][ C0] default_idle+0x9/0x10 [ 10.420509][ C0] default_idle_call+0x6e/0xb0 [ 10.420510][ C0] cpuidle_idle_call.constprop.0+0x237/0x410 [ 10.420512][ C0] ? arch_cpu_idle_exit+0x40/0x40 [ 10.420514][ C0] ? mark_tsc_async_resets+0x30/0x30 [ 10.420517][ C0] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 10.420518][ C0] ? rcu_is_watching+0x16/0xd0 [ 10.420519][ C0] do_idle+0xd8/0x190 [ 10.420521][ C0] cpu_startup_entry+0x53/0x70 [ 10.420523][ C0] rest_init+0x279/0x280 [ 10.420524][ C0] ? __cpuidle_text_end+0x3b/0x3b [ 10.420526][ C0] ? rest_init+0x280/0x280 [ 10.420528][ C0] ? acpi_hw_set_mode+0x4a0/0x4a0 [ 10.420530][ C0] ? cpus_read_unlock+0x4f/0xc0 [ 10.420532][ C0] ? acpi_enable+0x1e4/0x330 [ 10.420535][ C0] start_kernel+0x3af/0x3b0 [ 10.420536][ C0] x86_64_start_reservations+0x24/0x30 [ 10.420538][ C0] x86_64_start_kernel+0x12b/0x130 [ 10.420539][ C0] common_startup_64+0x13e/0x148 [ 10.420542][ C0] [ 10.420546][ C0] BUG: sleeping function called from invalid context at ./include/linux/sched/mm.h:322 [ 10.420548][ C0] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 0, name: swapper/0 [ 10.420549][ C0] preempt_count: 103, expected: 0 [ 10.420550][ C0] RCU nest depth: 0, expected: 0 [ 10.420551][ C0] INFO: lockdep is turned off. [ 10.420551][ C0] irq event stamp: 326667 [ 10.420552][ C0] hardirqs last enabled at (326666): [] __down_trylock_console_sem+0x86/0xa0 [ 10.420554][ C0] hardirqs last disabled at (326667): [] console_emit_next_record+0x3d4/0x4c0 [ 10.420556][ C0] softirqs last enabled at (326646): [] handle_softirqs+0x67c/0x900 [ 10.420557][ C0] softirqs last disabled at (326653): [] __irq_exit_rcu+0x145/0x1c0 [ 10.420559][ C0] Preemption disabled at: [ 10.420560][ C0] [] schedule_preempt_disabled+0x22/0x30 [ 10.420562][ C0] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 7.2.0-virtme #1 PREEMPT(full) [ 10.420564][ C0] Tainted: [W]=WARN [ 10.420565][ C0] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 10.420565][ C0] Call Trace: [ 10.420566][ C0] [ 10.420566][ C0] dump_stack_lvl+0x6f/0xa0 [ 10.420568][ C0] ? schedule_preempt_disabled+0x22/0x30 [ 10.420570][ C0] __might_resched.cold+0x1fe/0x2c1 [ 10.420573][ C0] ? perf_trace_sched_switch+0x7c0/0x7c0 [ 10.420575][ C0] ? rcu_is_watching+0x16/0xd0 [ 10.420576][ C0] __kmalloc_noprof+0x443/0x760 [ 10.420578][ C0] ? __rwlock_init+0x150/0x150 [ 10.420580][ C0] ? alloc_buf.isra.0+0x4b/0x260 [ 10.420582][ C0] ? do_raw_spin_unlock+0x59/0x250 [ 10.420584][ C0] alloc_buf.isra.0+0x4b/0x260 [ 10.420586][ C0] put_chars+0x1e1/0x2f0 [ 10.420588][ C0] ? __send_to_port+0x420/0x420 [ 10.420590][ C0] ? mark_lock+0x1d7/0xa00 [ 10.420592][ C0] hvc_console_print+0x292/0x780 [ 10.420595][ C0] ? hvc_write+0x3a0/0x3a0 [ 10.420596][ C0] ? rcu_is_watching+0x16/0xd0 [ 10.420597][ C0] ? rcu_is_watching+0x16/0xd0 [ 10.420599][ C0] console_emit_next_record+0x22f/0x4c0 [ 10.420601][ C0] ? devkmsg_read+0x4b0/0x4b0 [ 10.420602][ C0] ? console_flush_one_record+0x106/0x710 [ 10.420603][ C0] ? rcu_is_watching+0x16/0xd0 [ 10.420605][ C0] ? lock_acquire+0x13c/0x160 [ 10.420607][ C0] console_flush_one_record+0x46f/0x710 [ 10.420609][ C0] ? console_emit_next_record+0x4c0/0x4c0 [ 10.420610][ C0] ? __lock_acquire+0x518/0xc20 [ 10.420612][ C0] console_unlock+0xee/0x1f0 [ 10.420614][ C0] ? console_flush_one_record+0x710/0x710 [ 10.420615][ C0] ? rcu_is_watching+0x16/0xd0 [ 10.420616][ C0] ? lock_acquire+0x60/0x160 [ 10.420618][ C0] ? __down_trylock_console_sem+0x5e/0xa0 [ 10.420620][ C0] ? vprintk_emit+0x320/0x3e0 [ 10.420622][ C0] vprintk_emit+0x37c/0x3e0 [ 10.420623][ C0] ? wake_up_klogd_work_func+0x90/0x90 [ 10.420625][ C0] ? clocksource_watchdog.part.0+0x450/0x540 [ 10.420627][ C0] _printk+0xc7/0x100 [ 10.420629][ C0] ? snapshot_read.cold+0x21/0x21 [ 10.420631][ C0] ? clocksource_watchdog.part.0+0x450/0x540 [ 10.420632][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 10.420635][ C0] clocksource_watchdog.part.0.cold+0x105/0x1dd [ 10.420636][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 10.420638][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 10.420640][ C0] call_timer_fn+0x160/0x4d0 [ 10.420641][ C0] ? detach_if_pending+0x1d0/0x1d0 [ 10.420642][ C0] ? debug_object_active_state+0x430/0x430 [ 10.420644][ C0] ? find_held_lock+0x2b/0x80 [ 10.420646][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 10.420648][ C0] ? rcu_is_watching+0x16/0xd0 [ 10.420649][ C0] __run_timers+0x68f/0xaa0 [ 10.420651][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 10.420653][ C0] ? __bpf_trace_itimer_expire+0x10/0x10 [ 10.420654][ C0] ? __lock_acquire+0x518/0xc20 [ 10.420657][ C0] ? __rwlock_init+0x150/0x150 [ 10.420660][ C0] run_timer_softirq+0xf0/0x160 [ 10.420661][ C0] ? __run_timers+0xaa0/0xaa0 [ 10.420662][ C0] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 10.420663][ C0] ? rcu_is_watching+0x16/0xd0 [ 10.420664][ C0] handle_softirqs+0x1d3/0x900 [ 10.420667][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 10.420668][ C0] ? _local_bh_enable+0xc0/0xc0 [ 10.420670][ C0] __irq_exit_rcu+0x145/0x1c0 [ 10.420672][ C0] irq_exit_rcu+0xe/0x30 [ 10.420674][ C0] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 10.420676][ C0] [ 10.420676][ C0] [ 10.420677][ C0] ? lockdep_hardirqs_on_prepare.part.0+0x9a/0x160 [ 10.420679][ C0] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 10.420680][ C0] RIP: 0010:pv_native_safe_halt+0xf/0x10 [ 10.420682][ C0] Code: 48 8b 3d d4 41 10 02 e8 1f 00 00 00 48 2b 05 98 72 b2 00 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa eb 07 0f 00 2d 53 45 29 00 fb f4 0f 1f 40 d6 48 83 ec 20 8b 17 49 89 f8 83 e2 fe 41 89 d2 0f 01 [ 10.420683][ C0] RSP: 0018:ffffffff8b207cf8 EFLAGS: 00000296 [ 10.420684][ C0] RAX: 000000000004fbfb RBX: ffffffff8b21c600 RCX: ffffffff880f4ce7 [ 10.420685][ C0] RDX: ffffffff8b21c600 RSI: ffffffff8b088c54 RDI: ffffffff8aa93160 [ 10.420686][ C0] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 10.420687][ C0] R10: 0000000000000000 R11: 0000000000000001 R12: 1ffffffff1640fa2 [ 10.420687][ C0] R13: 0000000000000000 R14: dffffc0000000000 R15: 0000000000014770 [ 10.420689][ C0] ? cpuidle_idle_call.constprop.0+0x237/0x410 [ 10.420691][ C0] ? lockdep_hardirqs_on+0x91/0x130 [ 10.420693][ C0] default_idle+0x9/0x10 [ 10.420694][ C0] default_idle_call+0x6e/0xb0 [ 10.420695][ C0] cpuidle_idle_call.constprop.0+0x237/0x410 [ 10.420697][ C0] ? arch_cpu_idle_exit+0x40/0x40 [ 10.420699][ C0] ? mark_tsc_async_resets+0x30/0x30 [ 10.420700][ C0] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 10.420702][ C0] ? rcu_is_watching+0x16/0xd0 [ 10.420703][ C0] do_idle+0xd8/0x190 [ 10.420705][ C0] cpu_startup_entry+0x53/0x70 [ 10.420706][ C0] rest_init+0x279/0x280 [ 10.420708][ C0] ? __cpuidle_text_end+0x3b/0x3b [ 10.420710][ C0] ? rest_init+0x280/0x280 [ 10.420711][ C0] ? acpi_hw_set_mode+0x4a0/0x4a0 [ 10.420713][ C0] ? cpus_read_unlock+0x4f/0xc0 [ 10.420714][ C0] ? acpi_enable+0x1e4/0x330 [ 10.420716][ C0] start_kernel+0x3af/0x3b0 [ 10.420718][ C0] x86_64_start_reservations+0x24/0x30 [ 10.420719][ C0] x86_64_start_kernel+0x12b/0x130 [ 10.420721][ C0] common_startup_64+0x13e/0x148 [ 10.420724][ C0] [ 12.169019][ T471] nf_conntrack: nf_conntrack: table full in netns 4026532045, dropping packet [ 12.169036][ T471] BUG: sleeping function called from invalid context at ./include/linux/sched/mm.h:322 [ 12.169039][ T471] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 471, name: conntrack [ 12.169040][ T471] preempt_count: 1, expected: 0 [ 12.169041][ T471] RCU nest depth: 0, expected: 0 [ 12.169042][ T471] INFO: lockdep is turned off. [ 12.169043][ T471] irq event stamp: 0 [ 12.169044][ T471] hardirqs last enabled at (0): [<0000000000000000>] 0x0 [ 12.169049][ T471] hardirqs last disabled at (0): [] copy_process+0x1d01/0x62c0 [ 12.169056][ T471] softirqs last enabled at (0): [] copy_process+0x1d55/0x62c0 [ 12.169058][ T471] softirqs last disabled at (0): [<0000000000000000>] 0x0 [ 12.169059][ T471] Preemption disabled at: [ 12.169060][ T471] [] vprintk_emit+0x31b/0x3e0 [ 12.169067][ T471] CPU: 2 UID: 0 PID: 471 Comm: conntrack Tainted: G W 7.2.0-virtme #1 PREEMPT(full) [ 12.169070][ T471] Tainted: [W]=WARN [ 12.169071][ T471] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 12.169073][ T471] Call Trace: [ 12.169075][ T471] [ 12.169076][ T471] dump_stack_lvl+0x6f/0xa0 [ 12.169081][ T471] ? vprintk_emit+0x31b/0x3e0 [ 12.169082][ T471] __might_resched.cold+0x1fe/0x2c1 [ 12.169086][ T471] ? perf_trace_sched_switch+0x7c0/0x7c0 [ 12.169089][ T471] ? rcu_is_watching+0x16/0xd0 [ 12.169092][ T471] __kmalloc_noprof+0x443/0x760 [ 12.169095][ T471] ? __rwlock_init+0x150/0x150 [ 12.169098][ T471] ? alloc_buf.isra.0+0x4b/0x260 [ 12.169103][ T471] ? do_raw_spin_unlock+0x59/0x250 [ 12.169105][ T471] alloc_buf.isra.0+0x4b/0x260 [ 12.169107][ T471] put_chars+0x1e1/0x2f0 [ 12.169109][ T471] ? desc_read_finalized_seq+0x79/0x120 [ 12.169111][ T471] ? __send_to_port+0x420/0x420 [ 12.169113][ T471] ? console_prepend_replay+0x20/0x20 [ 12.169114][ T471] hvc_console_print+0x292/0x780 [ 12.169118][ T471] ? hvc_write+0x3a0/0x3a0 [ 12.169120][ T471] ? do_raw_spin_lock+0x131/0x280 [ 12.169122][ T471] ? __rwlock_init+0x150/0x150 [ 12.169123][ T471] ? rcu_is_watching+0x16/0xd0 [ 12.169124][ T471] ? rcu_is_watching+0x16/0xd0 [ 12.169126][ T471] ? lock_acquire+0x13c/0x160 [ 12.169128][ T471] console_emit_next_record+0x22f/0x4c0 [ 12.169130][ T471] ? devkmsg_read+0x4b0/0x4b0 [ 12.169132][ T471] ? rcu_is_watching+0x16/0xd0 [ 12.169133][ T471] ? rcu_is_watching+0x16/0xd0 [ 12.169134][ T471] ? rcu_is_watching+0x16/0xd0 [ 12.169135][ T471] ? lock_acquire+0x13c/0x160 [ 12.169137][ T471] ? console_flush_one_record+0x111/0x710 [ 12.169138][ T471] console_flush_one_record+0x46f/0x710 [ 12.169140][ T471] ? console_emit_next_record+0x4c0/0x4c0 [ 12.169142][ T471] console_unlock+0xee/0x1f0 [ 12.169143][ T471] ? lock_acquire+0x13c/0x160 [ 12.169145][ T471] ? console_flush_one_record+0x710/0x710 [ 12.169146][ T471] ? rcu_is_watching+0x16/0xd0 [ 12.169147][ T471] ? lock_release+0x184/0x1f0 [ 12.169148][ T471] ? lock_acquire+0x60/0x160 [ 12.169150][ T471] ? __rcu_read_lock+0x30/0x70 [ 12.169151][ T471] ? __down_trylock_console_sem+0x5e/0xa0 [ 12.169153][ T471] ? vprintk_emit+0x320/0x3e0 [ 12.169155][ T471] vprintk_emit+0x37c/0x3e0 [ 12.169157][ T471] ? wake_up_klogd_work_func+0x90/0x90 [ 12.169158][ T471] ? do_raw_spin_trylock+0xb6/0x180 [ 12.169160][ T471] ? rcu_is_watching+0x16/0xd0 [ 12.169161][ T471] ? rcu_is_watching+0x16/0xd0 [ 12.169163][ T471] ? lock_acquire+0x13c/0x160 [ 12.169165][ T471] _printk+0xc7/0x100 [ 12.169167][ T471] ? snapshot_read.cold+0x21/0x21 [ 12.169169][ T471] ? ___ratelimit+0x4e7/0xa00 [ 12.169173][ T471] ? nf_conntrack_find_get+0x260/0x260 [nf_conntrack] [ 12.169181][ T471] ? idr_get_free+0x830/0x830 [ 12.169183][ T471] ? lock_release+0x184/0x1f0 [ 12.169184][ T471] ? trace_irq_enable.constprop.0+0x30/0x160 [ 12.169187][ T471] __nf_conntrack_alloc.cold+0x5b/0x67 [nf_conntrack] [ 12.169194][ T471] ctnetlink_create_conntrack+0xbf/0x1250 [nf_conntrack_netlink] [ 12.169196][ T471] ? trace_hardirqs_on+0x36/0x40 [ 12.169198][ T471] ? nf_conntrack_find_get+0xd2/0x260 [nf_conntrack] [ 12.169203][ T471] ? ctnetlink_conntrack_event+0x1530/0x1530 [nf_conntrack_netlink] [ 12.169205][ T471] ? rcu_is_watching+0x16/0xd0 [ 12.169206][ T471] ? nf_ct_gc_expired.part.0+0x190/0x190 [nf_conntrack] [ 12.169210][ T471] ? rcu_is_watching+0x16/0xd0 [ 12.169212][ T471] ? rcu_is_watching+0x16/0xd0 [ 12.169213][ T471] ? lock_release+0x184/0x1f0 [ 12.169215][ T471] ctnetlink_new_conntrack+0x507/0x959 [nf_conntrack_netlink] [ 12.169217][ T471] ? ctnetlink_create_conntrack+0x1250/0x1250 [nf_conntrack_netlink] [ 12.169219][ T471] ? ww_mutex_lock+0x160/0x160 [ 12.169222][ T471] ? nla_get_range_signed+0x3d0/0x3d0 [ 12.169225][ T471] ? is_bpf_text_address+0x3e/0x110 [ 12.169228][ T471] ? is_bpf_text_address+0x72/0x110 [ 12.169231][ T471] ? __nla_validate_parse+0x20d/0x480 [ 12.169233][ T471] nfnetlink_rcv_msg+0x712/0xed0 [ 12.169238][ T471] ? nfnetlink_bind+0x370/0x370 [ 12.169240][ T471] ? unwind_next_frame+0x69b/0x1ea0 [ 12.169243][ T471] ? entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 12.169248][ T471] netlink_rcv_skb+0x14e/0x3a0 [ 12.169251][ T471] ? nfnetlink_bind+0x370/0x370 [ 12.169253][ T471] ? netlink_ack+0xcf0/0xcf0 [ 12.169255][ T471] ? cap_capable+0x1d7/0x3d0 [ 12.169257][ T471] ? __netlink_lookup+0xb9/0x6b0 [ 12.169258][ T471] ? __netlink_lookup+0x49e/0x6b0 [ 12.169260][ T471] nfnetlink_rcv+0x170/0x520 [ 12.169262][ T471] ? nfnetlink_rcv_batch+0x1510/0x1510 [ 12.169264][ T471] ? lock_release+0x184/0x1f0 [ 12.169266][ T471] ? netlink_deliver_tap+0xf8/0x330 [ 12.169267][ T471] ? netlink_deliver_tap+0x141/0x330 [ 12.169268][ T471] netlink_unicast+0x486/0x750 [ 12.169270][ T471] ? netlink_attachskb+0x810/0x810 [ 12.169272][ T471] ? exit_to_user_mode_loop+0x158/0x5a0 [ 12.169275][ T471] netlink_sendmsg+0x735/0xc60 [ 12.169277][ T471] ? netlink_unicast+0x750/0x750 [ 12.169278][ T471] ? lock_release+0x184/0x1f0 [ 12.169280][ T471] ? __might_fault+0x97/0x140 [ 12.169283][ T471] ? _copy_from_user+0x42/0x90 [ 12.169286][ T471] __sys_sendto+0x2a9/0x3e0 [ 12.169289][ T471] ? __ia32_sys_getpeername+0xd0/0xd0 [ 12.169294][ T471] __x64_sys_sendto+0xe4/0x1f0 [ 12.169295][ T471] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 12.169297][ T471] ? trace_hardirqs_off_finish+0x147/0x1a0 [ 12.169298][ T471] ? trace_hardirqs_on+0x36/0x40 [ 12.169299][ T471] do_syscall_64+0x10e/0x570 [ 12.169301][ T471] ? exc_page_fault+0xee/0x100 [ 12.169303][ T471] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 12.169305][ T471] RIP: 0033:0x7fee6f16b54e [ 12.169307][ T471] Code: 4d 89 d8 e8 b4 bd 00 00 4c 8b 5d f8 41 8b 93 08 03 00 00 59 5e 48 83 f8 fc 74 11 c9 c3 0f 1f 80 00 00 00 00 48 8b 45 10 0f 05 c3 83 e2 39 83 fa 08 75 e7 e8 03 ff ff ff 0f 1f 00 f3 0f 1e fa [ 12.169308][ T471] RSP: 002b:00007fff1bde2980 EFLAGS: 00000202 ORIG_RAX: 000000000000002c [ 12.169312][ T471] RAX: ffffffffffffffda RBX: 00007fff1bde39e0 RCX: 00007fee6f16b54e [ 12.169314][ T471] RDX: 000000000000008c RSI: 00007fff1bde39e0 RDI: 0000000000000005 [ 12.169315][ T471] RBP: 00007fff1bde2990 R08: 00007fee6f318980 R09: 000000000000000c [ 12.169316][ T471] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fff1bde39e0 [ 12.169317][ T471] R13: 0000556f0dfd38a0 R14: 0000000000000000 R15: 0000000000000002 [ 12.169321][ T471] [ 12.198561][ T478] nf_conntrack: nf_conntrack: table full in netns 4026532045, dropping packet [ 12.198952][ T482] nf_conntrack: nf_conntrack: table full in netns 4026532045, dropping packet [ 12.199206][ T490] nf_conntrack: nf_conntrack: table full in netns 4026532045, dropping packet [ 12.200018][ T485] nf_conntrack: nf_conntrack: table full in netns 4026532045, dropping packet [ 12.200545][ T488] nf_conntrack: nf_conntrack: table full in netns 4026532045, dropping packet [ 12.201135][ T495] nf_conntrack: nf_conntrack: table full in netns 4026532045, dropping packet [ 12.201590][ T493] nf_conntrack: nf_conntrack: table full in netns 4026532045, dropping packet [ 12.202420][ T494] nf_conntrack: nf_conntrack: table full in netns 4026532045, dropping packet [ 12.203490][ T471] conntrack (471) used greatest stack depth: 24232 bytes left [ 12.204701][ T489] nf_conntrack: nf_conntrack: table full in netns 4026532045, dropping packet