[ 11.901689][ C0] clocksource: Watchdog remote CPU 2 read timed out [ 11.901915][ C0] [ 11.901916][ C0] ======================================================== [ 11.901918][ C0] WARNING: possible irq lock inversion dependency detected [ 11.901920][ C0] 7.2.0-virtme #1 Tainted: G W [ 11.901922][ C0] -------------------------------------------------------- [ 11.901922][ C0] swapper/0/0 just changed the state of lock: [ 11.901924][ C0] ffffffffa5a99660 (console_owner){..-.}-{0:0}, at: console_lock_spinning_enable+0x40/0x60 [ 11.901936][ C0] but this lock took another, SOFTIRQ-unsafe lock in the past: [ 11.901937][ C0] (fs_reclaim){+.+.}-{0:0} [ 11.901939][ C0] [ 11.901939][ C0] [ 11.901939][ C0] and interrupts could create inverse lock ordering between them. [ 11.901939][ C0] [ 11.901939][ C0] [ 11.901939][ C0] other info that might help us debug this: [ 11.901940][ C0] Possible interrupt unsafe locking scenario: [ 11.901940][ C0] [ 11.901941][ C0] CPU0 CPU1 [ 11.901941][ C0] ---- ---- [ 11.901941][ C0] lock(fs_reclaim); [ 11.901942][ C0] local_irq_disable(); [ 11.901943][ C0] lock(console_owner); [ 11.901944][ C0] lock(fs_reclaim); [ 11.901945][ C0] [ 11.901945][ C0] lock(console_owner); [ 11.901946][ C0] [ 11.901946][ C0] *** DEADLOCK *** [ 11.901946][ C0] [ 11.901946][ C0] locks held by swapper/0/0: 4, last CPU#0: [ 11.901948][ C0] #0: ffa0000000007c90 ((&watchdog_timer)){+.-.}-{0:0}, at: call_timer_fn+0x110/0x4d0 [ 11.901953][ C0] #1: ffffffffa5bde8b8 (watchdog_lock){+.-.}-{3:3}, at: clocksource_watchdog+0x15/0x30 [ 11.901957][ C0] #2: ffffffffa5b79cc0 (console_lock){+.+.}-{0:0}, at: vprintk_emit+0x320/0x3e0 [ 11.901961][ C0] #3: ffffffffa5b79d38 (console_srcu){....}-{0:0}, at: console_flush_one_record+0x106/0x710 [ 11.901964][ C0] [ 11.901964][ C0] the shortest dependencies between 2nd lock and 1st lock: [ 11.901968][ C0] -> (fs_reclaim){+.+.}-{0:0} { [ 11.901971][ C0] HARDIRQ-ON-W at: [ 11.901972][ C0] __lock_acquire+0x388/0xc20 [ 11.901975][ C0] lock_acquire.part.0+0xd4/0x280 [ 11.901977][ C0] fs_reclaim_acquire+0xd5/0x120 [ 11.901979][ C0] __kmalloc_cache_noprof+0x6e/0x620 [ 11.901982][ C0] kthread_create_worker_on_node+0xea/0x210 [ 11.901985][ C0] workqueue_init+0x2a/0x680 [ 11.901988][ C0] kernel_init_freeable+0x2fe/0x630 [ 11.901991][ C0] kernel_init+0x21/0x150 [ 11.901995][ C0] ret_from_fork+0x474/0x6b0 [ 11.901998][ C0] ret_from_fork_asm+0x11/0x20 [ 11.902001][ C0] SOFTIRQ-ON-W at: [ 11.902001][ C0] __lock_acquire+0x388/0xc20 [ 11.902003][ C0] lock_acquire.part.0+0xd4/0x280 [ 11.902004][ C0] fs_reclaim_acquire+0xd5/0x120 [ 11.902005][ C0] __kmalloc_cache_noprof+0x6e/0x620 [ 11.902007][ C0] kthread_create_worker_on_node+0xea/0x210 [ 11.902008][ C0] workqueue_init+0x2a/0x680 [ 11.902009][ C0] kernel_init_freeable+0x2fe/0x630 [ 11.902011][ C0] kernel_init+0x21/0x150 [ 11.902012][ C0] ret_from_fork+0x474/0x6b0 [ 11.902013][ C0] ret_from_fork_asm+0x11/0x20 [ 11.902014][ C0] INITIAL USE at: [ 11.902015][ C0] __lock_acquire+0x388/0xc20 [ 11.902016][ C0] lock_acquire.part.0+0xd4/0x280 [ 11.902018][ C0] fs_reclaim_acquire+0xd5/0x120 [ 11.902019][ C0] __kmalloc_cache_noprof+0x6e/0x620 [ 11.902020][ C0] kthread_create_worker_on_node+0xea/0x210 [ 11.902022][ C0] workqueue_init+0x2a/0x680 [ 11.902023][ C0] kernel_init_freeable+0x2fe/0x630 [ 11.902024][ C0] kernel_init+0x21/0x150 [ 11.902025][ C0] ret_from_fork+0x474/0x6b0 [ 11.902027][ C0] ret_from_fork_asm+0x11/0x20 [ 11.902028][ C0] } [ 11.902028][ C0] ... key at: [] __fs_reclaim_map+0x0/0xbe0 [ 11.902032][ C0] ... acquired at: [ 11.902033][ C0] __lock_acquire+0x518/0xc20 [ 11.902034][ C0] lock_acquire.part.0+0xd4/0x280 [ 11.902035][ C0] fs_reclaim_acquire+0xd5/0x120 [ 11.902037][ C0] __kmalloc_noprof+0xd3/0x760 [ 11.902038][ C0] alloc_buf.isra.0+0x4b/0x260 [ 11.902041][ C0] put_chars+0x1e1/0x2f0 [ 11.902043][ C0] hvc_console_print+0x292/0x780 [ 11.902045][ C0] console_emit_next_record+0x22f/0x4c0 [ 11.902047][ C0] console_flush_one_record+0x46f/0x710 [ 11.902049][ C0] console_unlock+0xee/0x1f0 [ 11.902050][ C0] vprintk_emit+0x37c/0x3e0 [ 11.902052][ C0] dev_vprintk_emit+0x27f/0x2c0 [ 11.902055][ C0] dev_printk_emit+0xb9/0xee [ 11.902057][ C0] _dev_info+0xe2/0x116 [ 11.902058][ C0] __devm_rtc_register_device.cold+0xc3/0x3a8 [ 11.902061][ C0] cmos_do_probe+0x73b/0x98a [ 11.902062][ C0] platform_probe+0xfe/0x1f0 [ 11.902065][ C0] call_driver_probe+0x61/0x1c0 [ 11.902067][ C0] really_probe+0x199/0x760 [ 11.902069][ C0] __driver_probe_device+0x24f/0x440 [ 11.902070][ C0] driver_probe_device+0x4a/0xf0 [ 11.902072][ C0] __driver_attach+0x1b8/0x540 [ 11.902073][ C0] bus_for_each_dev+0x130/0x1e0 [ 11.902075][ C0] bus_add_driver+0x2c8/0x530 [ 11.902076][ C0] driver_register+0x1a3/0x390 [ 11.902078][ C0] __platform_driver_probe+0x13f/0x270 [ 11.902079][ C0] cmos_init+0x31/0x40 [ 11.902082][ C0] do_one_initcall+0x124/0x4f0 [ 11.902084][ C0] kernel_init_freeable+0x596/0x630 [ 11.902086][ C0] kernel_init+0x21/0x150 [ 11.902087][ C0] ret_from_fork+0x474/0x6b0 [ 11.902089][ C0] ret_from_fork_asm+0x11/0x20 [ 11.902090][ C0] [ 11.902090][ C0] -> (console_owner){..-.}-{0:0} { [ 11.902092][ C0] IN-SOFTIRQ-W at: [ 11.902093][ C0] __lock_acquire+0x388/0xc20 [ 11.902094][ C0] lock_acquire.part.0+0xd4/0x280 [ 11.902095][ C0] console_lock_spinning_enable+0x5c/0x60 [ 11.902097][ C0] console_emit_next_record+0x1d1/0x4c0 [ 11.902099][ C0] console_flush_one_record+0x46f/0x710 [ 11.902100][ C0] console_unlock+0xee/0x1f0 [ 11.902102][ C0] vprintk_emit+0x37c/0x3e0 [ 11.902103][ C0] _printk+0xc7/0x100 [ 11.902106][ C0] clocksource_watchdog.part.0.cold+0x105/0x1dd [ 11.902108][ C0] call_timer_fn+0x160/0x4d0 [ 11.902110][ C0] __run_timers+0x68f/0xaa0 [ 11.902112][ C0] run_timer_softirq+0xf0/0x160 [ 11.902113][ C0] handle_softirqs+0x1d3/0x900 [ 11.902116][ C0] __irq_exit_rcu+0x145/0x1c0 [ 11.902117][ C0] irq_exit_rcu+0xe/0x30 [ 11.902118][ C0] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 11.902120][ C0] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 11.902122][ C0] pv_native_safe_halt+0xf/0x10 [ 11.902123][ C0] default_idle+0x9/0x10 [ 11.902124][ C0] default_idle_call+0x6e/0xb0 [ 11.902126][ C0] cpuidle_idle_call.constprop.0+0x237/0x410 [ 11.902128][ C0] do_idle+0xd8/0x190 [ 11.902129][ C0] cpu_startup_entry+0x53/0x70 [ 11.902130][ C0] rest_init+0x279/0x280 [ 11.902132][ C0] start_kernel+0x3af/0x3b0 [ 11.902133][ C0] x86_64_start_reservations+0x24/0x30 [ 11.902135][ C0] x86_64_start_kernel+0x12b/0x130 [ 11.902137][ C0] common_startup_64+0x13e/0x148 [ 11.902139][ C0] INITIAL USE at: [ 11.902140][ C0] } [ 11.902141][ C0] ... key at: [] console_owner_dep_map+0x0/0x60 [ 11.902144][ C0] ... acquired at: [ 11.902145][ C0] mark_lock+0x1d7/0xa00 [ 11.902146][ C0] mark_usage+0x42/0x170 [ 11.902147][ C0] __lock_acquire+0x388/0xc20 [ 11.902149][ C0] lock_acquire.part.0+0xd4/0x280 [ 11.902150][ C0] console_lock_spinning_enable+0x5c/0x60 [ 11.902152][ C0] console_emit_next_record+0x1d1/0x4c0 [ 11.902153][ C0] console_flush_one_record+0x46f/0x710 [ 11.902155][ C0] console_unlock+0xee/0x1f0 [ 11.902156][ C0] vprintk_emit+0x37c/0x3e0 [ 11.902158][ C0] _printk+0xc7/0x100 [ 11.902159][ C0] clocksource_watchdog.part.0.cold+0x105/0x1dd [ 11.902160][ C0] call_timer_fn+0x160/0x4d0 [ 11.902162][ C0] __run_timers+0x68f/0xaa0 [ 11.902163][ C0] run_timer_softirq+0xf0/0x160 [ 11.902165][ C0] handle_softirqs+0x1d3/0x900 [ 11.902166][ C0] __irq_exit_rcu+0x145/0x1c0 [ 11.902167][ C0] irq_exit_rcu+0xe/0x30 [ 11.902168][ C0] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 11.902169][ C0] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 11.902170][ C0] pv_native_safe_halt+0xf/0x10 [ 11.902171][ C0] default_idle+0x9/0x10 [ 11.902173][ C0] default_idle_call+0x6e/0xb0 [ 11.902174][ C0] cpuidle_idle_call.constprop.0+0x237/0x410 [ 11.902175][ C0] do_idle+0xd8/0x190 [ 11.902176][ C0] cpu_startup_entry+0x53/0x70 [ 11.902177][ C0] rest_init+0x279/0x280 [ 11.902178][ C0] start_kernel+0x3af/0x3b0 [ 11.902180][ C0] x86_64_start_reservations+0x24/0x30 [ 11.902181][ C0] x86_64_start_kernel+0x12b/0x130 [ 11.902183][ C0] common_startup_64+0x13e/0x148 [ 11.902184][ C0] [ 11.902184][ C0] [ 11.902184][ C0] stack backtrace: [ 11.902187][ C0] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 7.2.0-virtme #1 PREEMPT(full) [ 11.902190][ C0] Tainted: [W]=WARN [ 11.902191][ C0] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 11.902193][ C0] Call Trace: [ 11.902194][ C0] [ 11.902195][ C0] dump_stack_lvl+0x6f/0xa0 [ 11.902199][ C0] print_irq_inversion_bug.part.0.cold+0xe6/0x143 [ 11.902202][ C0] mark_lock_irq+0x989/0x9c0 [ 11.902203][ C0] ? add_lock_to_list+0x2c/0x1a0 [ 11.902206][ C0] mark_lock+0x1d7/0xa00 [ 11.902208][ C0] mark_usage+0x42/0x170 [ 11.902209][ C0] __lock_acquire+0x388/0xc20 [ 11.902212][ C0] lock_acquire.part.0+0xd4/0x280 [ 11.902213][ C0] ? console_lock_spinning_enable+0x40/0x60 [ 11.902215][ C0] ? rcu_is_watching+0x16/0xd0 [ 11.902218][ C0] ? lock_acquire+0x13c/0x160 [ 11.902220][ C0] console_lock_spinning_enable+0x5c/0x60 [ 11.902222][ C0] ? console_lock_spinning_enable+0x40/0x60 [ 11.902224][ C0] console_emit_next_record+0x1d1/0x4c0 [ 11.902226][ C0] ? devkmsg_read+0x4b0/0x4b0 [ 11.902227][ C0] ? console_flush_one_record+0x106/0x710 [ 11.902229][ C0] ? rcu_is_watching+0x16/0xd0 [ 11.902231][ C0] ? lock_acquire+0x13c/0x160 [ 11.902233][ C0] console_flush_one_record+0x46f/0x710 [ 11.902235][ C0] ? console_emit_next_record+0x4c0/0x4c0 [ 11.902237][ C0] ? __lock_acquire+0x518/0xc20 [ 11.902239][ C0] console_unlock+0xee/0x1f0 [ 11.902241][ C0] ? console_flush_one_record+0x710/0x710 [ 11.902242][ C0] ? rcu_is_watching+0x16/0xd0 [ 11.902244][ C0] ? lock_acquire+0x60/0x160 [ 11.902246][ C0] ? __down_trylock_console_sem+0x5e/0xa0 [ 11.902248][ C0] ? vprintk_emit+0x320/0x3e0 [ 11.902250][ C0] vprintk_emit+0x37c/0x3e0 [ 11.902252][ C0] ? wake_up_klogd_work_func+0x90/0x90 [ 11.902254][ C0] ? clocksource_watchdog.part.0+0x450/0x540 [ 11.902256][ C0] _printk+0xc7/0x100 [ 11.902258][ C0] ? snapshot_read.cold+0x21/0x21 [ 11.902259][ C0] ? clocksource_watchdog.part.0+0x450/0x540 [ 11.902261][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 11.902263][ C0] clocksource_watchdog.part.0.cold+0x105/0x1dd [ 11.902265][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 11.902266][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 11.902268][ C0] call_timer_fn+0x160/0x4d0 [ 11.902270][ C0] ? detach_if_pending+0x1d0/0x1d0 [ 11.902272][ C0] ? debug_object_active_state+0x430/0x430 [ 11.902275][ C0] ? find_held_lock+0x2b/0x80 [ 11.902277][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 11.902279][ C0] ? rcu_is_watching+0x16/0xd0 [ 11.902281][ C0] __run_timers+0x68f/0xaa0 [ 11.902283][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 11.902285][ C0] ? __bpf_trace_itimer_expire+0x10/0x10 [ 11.902287][ C0] ? __lock_acquire+0x518/0xc20 [ 11.902289][ C0] ? __rwlock_init+0x150/0x150 [ 11.902292][ C0] run_timer_softirq+0xf0/0x160 [ 11.902294][ C0] ? __run_timers+0xaa0/0xaa0 [ 11.902295][ C0] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 11.902298][ C0] ? rcu_is_watching+0x16/0xd0 [ 11.902300][ C0] handle_softirqs+0x1d3/0x900 [ 11.902301][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 11.902303][ C0] ? _local_bh_enable+0xc0/0xc0 [ 11.902305][ C0] __irq_exit_rcu+0x145/0x1c0 [ 11.902306][ C0] irq_exit_rcu+0xe/0x30 [ 11.902308][ C0] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 11.902309][ C0] [ 11.902310][ C0] [ 11.902310][ C0] ? lockdep_hardirqs_on_prepare.part.0+0x9a/0x160 [ 11.902312][ C0] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 11.902313][ C0] RIP: 0010:pv_native_safe_halt+0xf/0x10 [ 11.902316][ C0] Code: 48 8b 3d 94 12 06 02 e8 1f 00 00 00 48 2b 05 58 13 a0 00 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa eb 07 0f 00 2d 13 b6 17 00 fb f4 0f 1f 40 d6 48 83 ec 20 8b 17 49 89 f8 83 e2 fe 41 89 d2 0f 01 [ 11.902317][ C0] RSP: 0018:ffffffffa5807cf8 EFLAGS: 00000296 [ 11.902319][ C0] RAX: 000000000005a2f7 RBX: ffffffffa582c600 RCX: ffffffffa26fa307 [ 11.902321][ C0] RDX: ffffffffa582c600 RSI: ffffffffa5678e7e RDI: ffffffffa508dce0 [ 11.902322][ C0] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 11.902322][ C0] R10: 0000000000000000 R11: 0000000000000001 R12: 1ffffffff4b00fa2 [ 11.902323][ C0] R13: 0000000000000000 R14: dffffc0000000000 R15: 0000000000014770 [ 11.902325][ C0] ? cpuidle_idle_call.constprop.0+0x237/0x410 [ 11.902327][ C0] ? lockdep_hardirqs_on+0x91/0x130 [ 11.902328][ C0] default_idle+0x9/0x10 [ 11.902329][ C0] default_idle_call+0x6e/0xb0 [ 11.902331][ C0] cpuidle_idle_call.constprop.0+0x237/0x410 [ 11.902332][ C0] ? arch_cpu_idle_exit+0x40/0x40 [ 11.902334][ C0] ? mark_tsc_async_resets+0x30/0x30 [ 11.902337][ C0] ? rcu_is_watching+0x16/0xd0 [ 11.902339][ C0] do_idle+0xd8/0x190 [ 11.902340][ C0] cpu_startup_entry+0x53/0x70 [ 11.902341][ C0] rest_init+0x279/0x280 [ 11.902343][ C0] ? __cpuidle_text_end+0x3b/0x3b [ 11.902345][ C0] ? rest_init+0x280/0x280 [ 11.902347][ C0] ? acpi_hw_set_mode+0x4a0/0x4a0 [ 11.902350][ C0] ? cpus_read_unlock+0x4f/0xc0 [ 11.902352][ C0] ? acpi_enable+0x1e4/0x330 [ 11.902355][ C0] start_kernel+0x3af/0x3b0 [ 11.902358][ C0] x86_64_start_reservations+0x24/0x30 [ 11.902359][ C0] x86_64_start_kernel+0x12b/0x130 [ 11.902361][ C0] common_startup_64+0x13e/0x148 [ 11.902364][ C0] [ 11.902368][ C0] BUG: sleeping function called from invalid context at ./include/linux/sched/mm.h:322 [ 11.902370][ C0] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 0, name: swapper/0 [ 11.902371][ C0] preempt_count: 103, expected: 0 [ 11.902372][ C0] RCU nest depth: 0, expected: 0 [ 11.902372][ C0] INFO: lockdep is turned off. [ 11.902373][ C0] irq event stamp: 369415 [ 11.902374][ C0] hardirqs last enabled at (369414): [] __down_trylock_console_sem+0x86/0xa0 [ 11.902376][ C0] hardirqs last disabled at (369415): [] console_emit_next_record+0x3d4/0x4c0 [ 11.902378][ C0] softirqs last enabled at (369386): [] handle_softirqs+0x67c/0x900 [ 11.902380][ C0] softirqs last disabled at (369401): [] __irq_exit_rcu+0x145/0x1c0 [ 11.902381][ C0] Preemption disabled at: [ 11.902382][ C0] [] schedule_preempt_disabled+0x22/0x30 [ 11.902385][ C0] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 7.2.0-virtme #1 PREEMPT(full) [ 11.902387][ C0] Tainted: [W]=WARN [ 11.902387][ C0] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 11.902388][ C0] Call Trace: [ 11.902388][ C0] [ 11.902389][ C0] dump_stack_lvl+0x6f/0xa0 [ 11.902391][ C0] ? schedule_preempt_disabled+0x22/0x30 [ 11.902392][ C0] __might_resched.cold+0x1fe/0x2c1 [ 11.902395][ C0] ? perf_trace_sched_switch+0x7c0/0x7c0 [ 11.902398][ C0] ? rcu_is_watching+0x16/0xd0 [ 11.902400][ C0] __kmalloc_noprof+0x443/0x760 [ 11.902402][ C0] ? __rwlock_init+0x150/0x150 [ 11.902403][ C0] ? alloc_buf.isra.0+0x4b/0x260 [ 11.902405][ C0] ? do_raw_spin_unlock+0x59/0x250 [ 11.902407][ C0] alloc_buf.isra.0+0x4b/0x260 [ 11.902409][ C0] put_chars+0x1e1/0x2f0 [ 11.902411][ C0] ? __send_to_port+0x420/0x420 [ 11.902413][ C0] ? mark_lock+0x1d7/0xa00 [ 11.902415][ C0] hvc_console_print+0x292/0x780 [ 11.902418][ C0] ? hvc_write+0x3a0/0x3a0 [ 11.902419][ C0] ? rcu_is_watching+0x16/0xd0 [ 11.902421][ C0] ? rcu_is_watching+0x16/0xd0 [ 11.902423][ C0] console_emit_next_record+0x22f/0x4c0 [ 11.902425][ C0] ? devkmsg_read+0x4b0/0x4b0 [ 11.902427][ C0] ? console_flush_one_record+0x106/0x710 [ 11.902429][ C0] ? rcu_is_watching+0x16/0xd0 [ 11.902431][ C0] ? lock_acquire+0x13c/0x160 [ 11.902433][ C0] console_flush_one_record+0x46f/0x710 [ 11.902435][ C0] ? console_emit_next_record+0x4c0/0x4c0 [ 11.902436][ C0] ? __lock_acquire+0x518/0xc20 [ 11.902438][ C0] console_unlock+0xee/0x1f0 [ 11.902440][ C0] ? console_flush_one_record+0x710/0x710 [ 11.902442][ C0] ? rcu_is_watching+0x16/0xd0 [ 11.902444][ C0] ? lock_acquire+0x60/0x160 [ 11.902445][ C0] ? __down_trylock_console_sem+0x5e/0xa0 [ 11.902447][ C0] ? vprintk_emit+0x320/0x3e0 [ 11.902449][ C0] vprintk_emit+0x37c/0x3e0 [ 11.902451][ C0] ? wake_up_klogd_work_func+0x90/0x90 [ 11.902453][ C0] ? clocksource_watchdog.part.0+0x450/0x540 [ 11.902455][ C0] _printk+0xc7/0x100 [ 11.902457][ C0] ? snapshot_read.cold+0x21/0x21 [ 11.902459][ C0] ? clocksource_watchdog.part.0+0x450/0x540 [ 11.902460][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 11.902462][ C0] clocksource_watchdog.part.0.cold+0x105/0x1dd [ 11.902464][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 11.902465][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 11.902467][ C0] call_timer_fn+0x160/0x4d0 [ 11.902469][ C0] ? detach_if_pending+0x1d0/0x1d0 [ 11.902471][ C0] ? debug_object_active_state+0x430/0x430 [ 11.902472][ C0] ? find_held_lock+0x2b/0x80 [ 11.902474][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 11.902476][ C0] ? rcu_is_watching+0x16/0xd0 [ 11.902478][ C0] __run_timers+0x68f/0xaa0 [ 11.902479][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 11.902482][ C0] ? __bpf_trace_itimer_expire+0x10/0x10 [ 11.902483][ C0] ? __lock_acquire+0x518/0xc20 [ 11.902486][ C0] ? __rwlock_init+0x150/0x150 [ 11.902488][ C0] run_timer_softirq+0xf0/0x160 [ 11.902490][ C0] ? __run_timers+0xaa0/0xaa0 [ 11.902492][ C0] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 11.902493][ C0] ? rcu_is_watching+0x16/0xd0 [ 11.902495][ C0] handle_softirqs+0x1d3/0x900 [ 11.902496][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 11.902498][ C0] ? _local_bh_enable+0xc0/0xc0 [ 11.902500][ C0] __irq_exit_rcu+0x145/0x1c0 [ 11.902501][ C0] irq_exit_rcu+0xe/0x30 [ 11.902503][ C0] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 11.902504][ C0] [ 11.902504][ C0] [ 11.902505][ C0] ? lockdep_hardirqs_on_prepare.part.0+0x9a/0x160 [ 11.902506][ C0] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 11.902508][ C0] RIP: 0010:pv_native_safe_halt+0xf/0x10 [ 11.902509][ C0] Code: 48 8b 3d 94 12 06 02 e8 1f 00 00 00 48 2b 05 58 13 a0 00 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa eb 07 0f 00 2d 13 b6 17 00 fb f4 0f 1f 40 d6 48 83 ec 20 8b 17 49 89 f8 83 e2 fe 41 89 d2 0f 01 [ 11.902510][ C0] RSP: 0018:ffffffffa5807cf8 EFLAGS: 00000296 [ 11.902511][ C0] RAX: 000000000005a2f7 RBX: ffffffffa582c600 RCX: ffffffffa26fa307 [ 11.902512][ C0] RDX: ffffffffa582c600 RSI: ffffffffa5678e7e RDI: ffffffffa508dce0 [ 11.902513][ C0] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 11.902514][ C0] R10: 0000000000000000 R11: 0000000000000001 R12: 1ffffffff4b00fa2 [ 11.902514][ C0] R13: 0000000000000000 R14: dffffc0000000000 R15: 0000000000014770 [ 11.902516][ C0] ? cpuidle_idle_call.constprop.0+0x237/0x410 [ 11.902518][ C0] ? lockdep_hardirqs_on+0x91/0x130 [ 11.902519][ C0] default_idle+0x9/0x10 [ 11.902520][ C0] default_idle_call+0x6e/0xb0 [ 11.902522][ C0] cpuidle_idle_call.constprop.0+0x237/0x410 [ 11.902523][ C0] ? arch_cpu_idle_exit+0x40/0x40 [ 11.902525][ C0] ? mark_tsc_async_resets+0x30/0x30 [ 11.902526][ C0] ? rcu_is_watching+0x16/0xd0 [ 11.902528][ C0] do_idle+0xd8/0x190 [ 11.902529][ C0] cpu_startup_entry+0x53/0x70 [ 11.902531][ C0] rest_init+0x279/0x280 [ 11.902532][ C0] ? __cpuidle_text_end+0x3b/0x3b [ 11.902535][ C0] ? rest_init+0x280/0x280 [ 11.902536][ C0] ? acpi_hw_set_mode+0x4a0/0x4a0 [ 11.902538][ C0] ? cpus_read_unlock+0x4f/0xc0 [ 11.902539][ C0] ? acpi_enable+0x1e4/0x330 [ 11.902541][ C0] start_kernel+0x3af/0x3b0 [ 11.902544][ C0] x86_64_start_reservations+0x24/0x30 [ 11.902545][ C0] x86_64_start_kernel+0x12b/0x130 [ 11.902547][ C0] common_startup_64+0x13e/0x148 [ 11.902550][ C0] [ 17.951662][ T12] RDS: shutdown called in state 1 [ 17.951677][ T12] BUG: sleeping function called from invalid context at ./include/linux/sched/mm.h:322 [ 17.951680][ T12] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 12, name: kworker/u16:0 [ 17.951682][ T12] preempt_count: 1, expected: 0 [ 17.951683][ T12] RCU nest depth: 0, expected: 0 [ 17.951683][ T12] INFO: lockdep is turned off. [ 17.951684][ T12] irq event stamp: 13760 [ 17.951685][ T12] hardirqs last enabled at (13759): [] _raw_spin_unlock_irq+0x28/0x50 [ 17.951694][ T12] hardirqs last disabled at (13760): [] __schedule+0x1246/0x20e0 [ 17.951697][ T12] softirqs last enabled at (13752): [] rds_send_xmit+0x1198/0x26e0 [ 17.951700][ T12] softirqs last disabled at (13750): [] release_sock+0x21/0x240 [ 17.951703][ T12] Preemption disabled at: [ 17.951704][ T12] [] vprintk_emit+0x31b/0x3e0 [ 17.951713][ T12] CPU: 0 UID: 0 PID: 12 Comm: kworker/u16:0 Tainted: G W 7.2.0-virtme #1 PREEMPT(full) [ 17.951716][ T12] Tainted: [W]=WARN [ 17.951717][ T12] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 17.951719][ T12] Workqueue: krds_cp_wq#1/6 rds_shutdown_worker [ 17.951731][ T12] Call Trace: [ 17.951733][ T12] [ 17.951734][ T12] dump_stack_lvl+0x6f/0xa0 [ 17.951740][ T12] ? vprintk_emit+0x31b/0x3e0 [ 17.951741][ T12] __might_resched.cold+0x1fe/0x2c1 [ 17.951745][ T12] ? perf_trace_sched_switch+0x7c0/0x7c0 [ 17.951748][ T12] ? prb_next_reserve_seq+0x2d0/0x2d0 [ 17.951750][ T12] ? rcu_is_watching+0x16/0xd0 [ 17.951754][ T12] __kmalloc_noprof+0x443/0x760 [ 17.951758][ T12] ? __rwlock_init+0x150/0x150 [ 17.951760][ T12] ? alloc_buf.isra.0+0x4b/0x260 [ 17.951764][ T12] ? do_raw_spin_unlock+0x59/0x250 [ 17.951766][ T12] alloc_buf.isra.0+0x4b/0x260 [ 17.951768][ T12] put_chars+0x1e1/0x2f0 [ 17.951770][ T12] ? prb_final_commit+0x50/0x50 [ 17.951771][ T12] ? __send_to_port+0x420/0x420 [ 17.951773][ T12] ? printk_get_next_message+0x240/0x650 [ 17.951775][ T12] hvc_console_print+0x292/0x780 [ 17.951779][ T12] ? hvc_write+0x3a0/0x3a0 [ 17.951780][ T12] ? do_raw_spin_lock+0x131/0x280 [ 17.951782][ T12] ? vsnprintf+0x668/0x1700 [ 17.951784][ T12] ? __rwlock_init+0x150/0x150 [ 17.951785][ T12] ? rcu_is_watching+0x16/0xd0 [ 17.951787][ T12] ? lock_acquire+0x13c/0x160 [ 17.951789][ T12] console_emit_next_record+0x22f/0x4c0 [ 17.951792][ T12] ? devkmsg_read+0x4b0/0x4b0 [ 17.951794][ T12] ? rcu_is_watching+0x16/0xd0 [ 17.951795][ T12] ? rcu_is_watching+0x16/0xd0 [ 17.951797][ T12] ? rcu_is_watching+0x16/0xd0 [ 17.951799][ T12] ? lock_acquire+0x13c/0x160 [ 17.951800][ T12] ? console_flush_one_record+0x111/0x710 [ 17.951802][ T12] console_flush_one_record+0x46f/0x710 [ 17.951804][ T12] ? console_emit_next_record+0x4c0/0x4c0 [ 17.951807][ T12] console_unlock+0xee/0x1f0 [ 17.951809][ T12] ? lock_acquire+0x13c/0x160 [ 17.951810][ T12] ? console_flush_one_record+0x710/0x710 [ 17.951812][ T12] ? rcu_is_watching+0x16/0xd0 [ 17.951813][ T12] ? lock_release+0x184/0x1f0 [ 17.951815][ T12] ? lock_acquire+0x60/0x160 [ 17.951816][ T12] ? __rcu_read_lock+0x30/0x70 [ 17.951818][ T12] ? __down_trylock_console_sem+0x5e/0xa0 [ 17.951820][ T12] ? vprintk_emit+0x320/0x3e0 [ 17.951822][ T12] vprintk_emit+0x37c/0x3e0 [ 17.951824][ T12] ? wake_up_klogd_work_func+0x90/0x90 [ 17.951826][ T12] ? ww_mutex_lock+0x160/0x160 [ 17.951829][ T12] __rds_conn_path_error+0xbd/0x100 [ 17.951831][ T12] ? rds_check_all_paths+0xb0/0xb0 [ 17.951833][ T12] ? lock_acquire+0x13c/0x160 [ 17.951835][ T12] ? rcu_is_watching+0x16/0xd0 [ 17.951837][ T12] ? rds_conn_shutdown+0x652/0xa20 [ 17.951838][ T12] rds_conn_shutdown+0x695/0xa20 [ 17.951840][ T12] ? __rds_conn_path_error+0x100/0x100 [ 17.951841][ T12] ? debug_object_active_state+0x430/0x430 [ 17.951845][ T12] ? rcu_is_watching+0x16/0xd0 [ 17.951847][ T12] ? rcu_is_watching+0x16/0xd0 [ 17.951849][ T12] ? lock_acquire+0x13c/0x160 [ 17.951850][ T12] ? rcu_is_watching+0x16/0xd0 [ 17.951852][ T12] ? process_one_work+0xdfc/0x1560 [ 17.951855][ T12] process_one_work+0xe3e/0x1560 [ 17.951858][ T12] ? __queue_delayed_work+0x470/0x470 [ 17.951860][ T12] ? __rwlock_init+0x150/0x150 [ 17.951863][ T12] worker_thread+0x4f1/0xd60 [ 17.951866][ T12] ? rescuer_thread+0x1340/0x1340 [ 17.951867][ T12] kthread+0x367/0x460 [ 17.951870][ T12] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 17.951872][ T12] ? kthread_affine_preferred+0x4c0/0x4c0 [ 17.951874][ T12] ret_from_fork+0x474/0x6b0 [ 17.951877][ T12] ? arch_exit_to_user_mode_prepare.isra.0+0x120/0x120 [ 17.951879][ T12] ? __switch_to+0x5a3/0xe00 [ 17.951881][ T12] ? kthread_affine_preferred+0x4c0/0x4c0 [ 17.951883][ T12] ret_from_fork_asm+0x11/0x20 [ 17.951887][ T12] [ 25.974385][ T236] RDS: shutdown called in state 1 [ 25.974399][ T236] BUG: sleeping function called from invalid context at ./include/linux/sched/mm.h:322 [ 25.974402][ T236] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 236, name: kworker/u16:4 [ 25.974403][ T236] preempt_count: 1, expected: 0 [ 25.974404][ T236] RCU nest depth: 0, expected: 0 [ 25.974405][ T236] INFO: lockdep is turned off. [ 25.974406][ T236] irq event stamp: 6942 [ 25.974407][ T236] hardirqs last enabled at (6941): [] _raw_spin_unlock_irq+0x28/0x50 [ 25.974416][ T236] hardirqs last disabled at (6942): [] __schedule+0x1246/0x20e0 [ 25.974418][ T236] softirqs last enabled at (6938): [] rds_send_xmit+0x1198/0x26e0 [ 25.974421][ T236] softirqs last disabled at (6936): [] release_sock+0x21/0x240 [ 25.974425][ T236] Preemption disabled at: [ 25.974426][ T236] [] vprintk_emit+0x31b/0x3e0 [ 25.974434][ T236] CPU: 3 UID: 0 PID: 236 Comm: kworker/u16:4 Tainted: G W 7.2.0-virtme #1 PREEMPT(full) [ 25.974438][ T236] Tainted: [W]=WARN [ 25.974439][ T236] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 25.974441][ T236] Workqueue: krds_cp_wq#1/5 rds_shutdown_worker [ 25.974446][ T236] Call Trace: [ 25.974447][ T236] [ 25.974449][ T236] dump_stack_lvl+0x6f/0xa0 [ 25.974455][ T236] ? vprintk_emit+0x31b/0x3e0 [ 25.974457][ T236] __might_resched.cold+0x1fe/0x2c1 [ 25.974461][ T236] ? perf_trace_sched_switch+0x7c0/0x7c0 [ 25.974464][ T236] ? prb_next_reserve_seq+0x2d0/0x2d0 [ 25.974466][ T236] ? rcu_is_watching+0x16/0xd0 [ 25.974470][ T236] __kmalloc_noprof+0x443/0x760 [ 25.974473][ T236] ? __rwlock_init+0x150/0x150 [ 25.974476][ T236] ? alloc_buf.isra.0+0x4b/0x260 [ 25.974480][ T236] ? do_raw_spin_unlock+0x59/0x250 [ 25.974482][ T236] alloc_buf.isra.0+0x4b/0x260 [ 25.974484][ T236] put_chars+0x1e1/0x2f0 [ 25.974486][ T236] ? prb_final_commit+0x50/0x50 [ 25.974487][ T236] ? __send_to_port+0x420/0x420 [ 25.974489][ T236] ? printk_get_next_message+0x240/0x650 [ 25.974492][ T236] hvc_console_print+0x292/0x780 [ 25.974495][ T236] ? hvc_write+0x3a0/0x3a0 [ 25.974497][ T236] ? do_raw_spin_lock+0x131/0x280 [ 25.974498][ T236] ? vsnprintf+0x668/0x1700 [ 25.974500][ T236] ? __rwlock_init+0x150/0x150 [ 25.974502][ T236] ? rcu_is_watching+0x16/0xd0 [ 25.974504][ T236] ? lock_acquire+0x13c/0x160 [ 25.974506][ T236] console_emit_next_record+0x22f/0x4c0 [ 25.974509][ T236] ? devkmsg_read+0x4b0/0x4b0 [ 25.974511][ T236] ? rcu_is_watching+0x16/0xd0 [ 25.974513][ T236] ? rcu_is_watching+0x16/0xd0 [ 25.974514][ T236] ? rcu_is_watching+0x16/0xd0 [ 25.974516][ T236] ? lock_acquire+0x13c/0x160 [ 25.974518][ T236] ? console_flush_one_record+0x111/0x710 [ 25.974520][ T236] console_flush_one_record+0x46f/0x710 [ 25.974522][ T236] ? console_emit_next_record+0x4c0/0x4c0 [ 25.974525][ T236] console_unlock+0xee/0x1f0 [ 25.974526][ T236] ? lock_acquire+0x13c/0x160 [ 25.974528][ T236] ? console_flush_one_record+0x710/0x710 [ 25.974530][ T236] ? rcu_is_watching+0x16/0xd0 [ 25.974531][ T236] ? lock_release+0x184/0x1f0 [ 25.974533][ T236] ? lock_acquire+0x60/0x160 [ 25.974534][ T236] ? __rcu_read_lock+0x30/0x70 [ 25.974536][ T236] ? __down_trylock_console_sem+0x5e/0xa0 [ 25.974538][ T236] ? vprintk_emit+0x320/0x3e0 [ 25.974540][ T236] vprintk_emit+0x37c/0x3e0 [ 25.974543][ T236] ? wake_up_klogd_work_func+0x90/0x90 [ 25.974545][ T236] ? ww_mutex_lock+0x160/0x160 [ 25.974547][ T236] __rds_conn_path_error+0xbd/0x100 [ 25.974550][ T236] ? rds_check_all_paths+0xb0/0xb0 [ 25.974555][ T236] ? lock_acquire+0x13c/0x160 [ 25.974556][ T236] ? rcu_is_watching+0x16/0xd0 [ 25.974558][ T236] ? rds_conn_shutdown+0x652/0xa20 [ 25.974560][ T236] rds_conn_shutdown+0x695/0xa20 [ 25.974562][ T236] ? __rds_conn_path_error+0x100/0x100 [ 25.974563][ T236] ? debug_object_active_state+0x430/0x430 [ 25.974567][ T236] ? rcu_is_watching+0x16/0xd0 [ 25.974569][ T236] ? rcu_is_watching+0x16/0xd0 [ 25.974571][ T236] ? rcu_is_watching+0x16/0xd0 [ 25.974572][ T236] ? lock_acquire+0x13c/0x160 [ 25.974574][ T236] ? rcu_is_watching+0x16/0xd0 [ 25.974576][ T236] ? process_one_work+0xdfc/0x1560 [ 25.974579][ T236] process_one_work+0xe3e/0x1560 [ 25.974581][ T236] ? __queue_delayed_work+0x470/0x470 [ 25.974584][ T236] ? __rwlock_init+0x150/0x150 [ 25.974586][ T236] worker_thread+0x4f1/0xd60 [ 25.974589][ T236] ? rescuer_thread+0x1340/0x1340 [ 25.974591][ T236] kthread+0x367/0x460 [ 25.974594][ T236] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 25.974596][ T236] ? kthread_affine_preferred+0x4c0/0x4c0 [ 25.974598][ T236] ret_from_fork+0x474/0x6b0 [ 25.974601][ T236] ? arch_exit_to_user_mode_prepare.isra.0+0x120/0x120 [ 25.974603][ T236] ? __switch_to+0x5a3/0xe00 [ 25.974605][ T236] ? kthread_affine_preferred+0x4c0/0x4c0 [ 25.974607][ T236] ret_from_fork_asm+0x11/0x20 [ 25.974610][ T236]