[ 15.923762][ C0] clocksource: Watchdog remote CPU 2 read timed out [ 15.923806][ C0] [ 15.923807][ C0] ======================================================== [ 15.923808][ C0] WARNING: possible irq lock inversion dependency detected [ 15.923810][ C0] 7.2.0-virtme #1 Tainted: G W [ 15.923812][ C0] -------------------------------------------------------- [ 15.923813][ C0] swapper/0/0 just changed the state of lock: [ 15.923814][ C0] ffffffff95c99660 (console_owner){..-.}-{0:0}, at: console_lock_spinning_enable+0x40/0x60 [ 15.923825][ C0] but this lock took another, SOFTIRQ-unsafe lock in the past: [ 15.923826][ C0] (fs_reclaim){+.+.}-{0:0} [ 15.923828][ C0] [ 15.923828][ C0] [ 15.923828][ C0] and interrupts could create inverse lock ordering between them. [ 15.923828][ C0] [ 15.923829][ C0] [ 15.923829][ C0] other info that might help us debug this: [ 15.923829][ C0] Possible interrupt unsafe locking scenario: [ 15.923829][ C0] [ 15.923830][ C0] CPU0 CPU1 [ 15.923831][ C0] ---- ---- [ 15.923831][ C0] lock(fs_reclaim); [ 15.923832][ C0] local_irq_disable(); [ 15.923833][ C0] lock(console_owner); [ 15.923833][ C0] lock(fs_reclaim); [ 15.923834][ C0] [ 15.923835][ C0] lock(console_owner); [ 15.923836][ C0] [ 15.923836][ C0] *** DEADLOCK *** [ 15.923836][ C0] [ 15.923836][ C0] locks held by swapper/0/0: 4, last CPU#0: [ 15.923837][ C0] #0: ffa0000000007c90 ((&watchdog_timer)){+.-.}-{0:0}, at: call_timer_fn+0x110/0x4d0 [ 15.923843][ C0] #1: ffffffff95dde8b8 (watchdog_lock){+.-.}-{3:3}, at: clocksource_watchdog+0x15/0x30 [ 15.923847][ C0] #2: ffffffff95d79cc0 (console_lock){+.+.}-{0:0}, at: vprintk_emit+0x320/0x3e0 [ 15.923851][ C0] #3: ffffffff95d79d38 (console_srcu){....}-{0:0}, at: console_flush_one_record+0x106/0x710 [ 15.923854][ C0] [ 15.923854][ C0] the shortest dependencies between 2nd lock and 1st lock: [ 15.923858][ C0] -> (fs_reclaim){+.+.}-{0:0} { [ 15.923860][ C0] HARDIRQ-ON-W at: [ 15.923862][ C0] __lock_acquire+0x388/0xc20 [ 15.923864][ C0] lock_acquire.part.0+0xd4/0x280 [ 15.923866][ C0] fs_reclaim_acquire+0xd5/0x120 [ 15.923869][ C0] __kmalloc_cache_noprof+0x6e/0x620 [ 15.923871][ C0] kthread_create_worker_on_node+0xea/0x210 [ 15.923874][ C0] workqueue_init+0x2a/0x680 [ 15.923877][ C0] kernel_init_freeable+0x2fe/0x630 [ 15.923880][ C0] kernel_init+0x21/0x150 [ 15.923883][ C0] ret_from_fork+0x474/0x6b0 [ 15.923886][ C0] ret_from_fork_asm+0x11/0x20 [ 15.923889][ C0] SOFTIRQ-ON-W at: [ 15.923890][ C0] __lock_acquire+0x388/0xc20 [ 15.923891][ C0] lock_acquire.part.0+0xd4/0x280 [ 15.923893][ C0] fs_reclaim_acquire+0xd5/0x120 [ 15.923894][ C0] __kmalloc_cache_noprof+0x6e/0x620 [ 15.923895][ C0] kthread_create_worker_on_node+0xea/0x210 [ 15.923897][ C0] workqueue_init+0x2a/0x680 [ 15.923898][ C0] kernel_init_freeable+0x2fe/0x630 [ 15.923899][ C0] kernel_init+0x21/0x150 [ 15.923901][ C0] ret_from_fork+0x474/0x6b0 [ 15.923902][ C0] ret_from_fork_asm+0x11/0x20 [ 15.923903][ C0] INITIAL USE at: [ 15.923904][ C0] __lock_acquire+0x388/0xc20 [ 15.923905][ C0] lock_acquire.part.0+0xd4/0x280 [ 15.923906][ C0] fs_reclaim_acquire+0xd5/0x120 [ 15.923907][ C0] __kmalloc_cache_noprof+0x6e/0x620 [ 15.923909][ C0] kthread_create_worker_on_node+0xea/0x210 [ 15.923910][ C0] workqueue_init+0x2a/0x680 [ 15.923911][ C0] kernel_init_freeable+0x2fe/0x630 [ 15.923913][ C0] kernel_init+0x21/0x150 [ 15.923914][ C0] ret_from_fork+0x474/0x6b0 [ 15.923915][ C0] ret_from_fork_asm+0x11/0x20 [ 15.923917][ C0] } [ 15.923917][ C0] ... key at: [] __fs_reclaim_map+0x0/0xbe0 [ 15.923921][ C0] ... acquired at: [ 15.923921][ C0] __lock_acquire+0x518/0xc20 [ 15.923923][ C0] lock_acquire.part.0+0xd4/0x280 [ 15.923924][ C0] fs_reclaim_acquire+0xd5/0x120 [ 15.923925][ C0] __kmalloc_noprof+0xd3/0x760 [ 15.923927][ C0] alloc_buf.isra.0+0x4b/0x260 [ 15.923930][ C0] put_chars+0x1e1/0x2f0 [ 15.923932][ C0] hvc_console_print+0x292/0x780 [ 15.923934][ C0] console_emit_next_record+0x22f/0x4c0 [ 15.923935][ C0] console_flush_one_record+0x46f/0x710 [ 15.923937][ C0] console_unlock+0xee/0x1f0 [ 15.923939][ C0] vprintk_emit+0x37c/0x3e0 [ 15.923940][ C0] dev_vprintk_emit+0x27f/0x2c0 [ 15.923944][ C0] dev_printk_emit+0xb9/0xee [ 15.923945][ C0] _dev_info+0xe2/0x116 [ 15.923947][ C0] cmos_do_probe+0x8be/0x98a [ 15.923949][ C0] platform_probe+0xfe/0x1f0 [ 15.923952][ C0] call_driver_probe+0x61/0x1c0 [ 15.923954][ C0] really_probe+0x199/0x760 [ 15.923955][ C0] __driver_probe_device+0x24f/0x440 [ 15.923957][ C0] driver_probe_device+0x4a/0xf0 [ 15.923959][ C0] __driver_attach+0x1b8/0x540 [ 15.923960][ C0] bus_for_each_dev+0x130/0x1e0 [ 15.923961][ C0] bus_add_driver+0x2c8/0x530 [ 15.923963][ C0] driver_register+0x1a3/0x390 [ 15.923964][ C0] __platform_driver_probe+0x13f/0x270 [ 15.923965][ C0] cmos_init+0x31/0x40 [ 15.923968][ C0] do_one_initcall+0x124/0x4f0 [ 15.923970][ C0] kernel_init_freeable+0x596/0x630 [ 15.923972][ C0] kernel_init+0x21/0x150 [ 15.923973][ C0] ret_from_fork+0x474/0x6b0 [ 15.923974][ C0] ret_from_fork_asm+0x11/0x20 [ 15.923975][ C0] [ 15.923975][ C0] -> (console_owner){..-.}-{0:0} { [ 15.923977][ C0] IN-SOFTIRQ-W at: [ 15.923978][ C0] __lock_acquire+0x388/0xc20 [ 15.923979][ C0] lock_acquire.part.0+0xd4/0x280 [ 15.923981][ C0] console_lock_spinning_enable+0x5c/0x60 [ 15.923982][ C0] console_emit_next_record+0x1d1/0x4c0 [ 15.923984][ C0] console_flush_one_record+0x46f/0x710 [ 15.923985][ C0] console_unlock+0xee/0x1f0 [ 15.923987][ C0] vprintk_emit+0x37c/0x3e0 [ 15.923988][ C0] _printk+0xc7/0x100 [ 15.923991][ C0] clocksource_watchdog.part.0.cold+0x105/0x1dd [ 15.923994][ C0] call_timer_fn+0x160/0x4d0 [ 15.923995][ C0] __run_timers+0x68f/0xaa0 [ 15.923997][ C0] run_timer_softirq+0xf0/0x160 [ 15.923998][ C0] handle_softirqs+0x1d3/0x900 [ 15.924001][ C0] __irq_exit_rcu+0x145/0x1c0 [ 15.924002][ C0] irq_exit_rcu+0xe/0x30 [ 15.924003][ C0] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 15.924005][ C0] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 15.924007][ C0] pv_native_safe_halt+0xf/0x10 [ 15.924008][ C0] default_idle+0x9/0x10 [ 15.924009][ C0] default_idle_call+0x6e/0xb0 [ 15.924011][ C0] cpuidle_idle_call.constprop.0+0x237/0x410 [ 15.924013][ C0] do_idle+0xd8/0x190 [ 15.924014][ C0] cpu_startup_entry+0x53/0x70 [ 15.924016][ C0] rest_init+0x279/0x280 [ 15.924017][ C0] start_kernel+0x3af/0x3b0 [ 15.924019][ C0] x86_64_start_reservations+0x24/0x30 [ 15.924021][ C0] x86_64_start_kernel+0x12b/0x130 [ 15.924022][ C0] common_startup_64+0x13e/0x148 [ 15.924024][ C0] INITIAL USE at: [ 15.924025][ C0] } [ 15.924026][ C0] ... key at: [] console_owner_dep_map+0x0/0x60 [ 15.924029][ C0] ... acquired at: [ 15.924030][ C0] mark_lock+0x1d7/0xa00 [ 15.924031][ C0] mark_usage+0x42/0x170 [ 15.924032][ C0] __lock_acquire+0x388/0xc20 [ 15.924033][ C0] lock_acquire.part.0+0xd4/0x280 [ 15.924035][ C0] console_lock_spinning_enable+0x5c/0x60 [ 15.924036][ C0] console_emit_next_record+0x1d1/0x4c0 [ 15.924038][ C0] console_flush_one_record+0x46f/0x710 [ 15.924039][ C0] console_unlock+0xee/0x1f0 [ 15.924041][ C0] vprintk_emit+0x37c/0x3e0 [ 15.924042][ C0] _printk+0xc7/0x100 [ 15.924044][ C0] clocksource_watchdog.part.0.cold+0x105/0x1dd [ 15.924045][ C0] call_timer_fn+0x160/0x4d0 [ 15.924046][ C0] __run_timers+0x68f/0xaa0 [ 15.924048][ C0] run_timer_softirq+0xf0/0x160 [ 15.924049][ C0] handle_softirqs+0x1d3/0x900 [ 15.924051][ C0] __irq_exit_rcu+0x145/0x1c0 [ 15.924052][ C0] irq_exit_rcu+0xe/0x30 [ 15.924053][ C0] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 15.924054][ C0] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 15.924055][ C0] pv_native_safe_halt+0xf/0x10 [ 15.924056][ C0] default_idle+0x9/0x10 [ 15.924057][ C0] default_idle_call+0x6e/0xb0 [ 15.924058][ C0] cpuidle_idle_call.constprop.0+0x237/0x410 [ 15.924060][ C0] do_idle+0xd8/0x190 [ 15.924061][ C0] cpu_startup_entry+0x53/0x70 [ 15.924062][ C0] rest_init+0x279/0x280 [ 15.924063][ C0] start_kernel+0x3af/0x3b0 [ 15.924065][ C0] x86_64_start_reservations+0x24/0x30 [ 15.924066][ C0] x86_64_start_kernel+0x12b/0x130 [ 15.924068][ C0] common_startup_64+0x13e/0x148 [ 15.924069][ C0] [ 15.924069][ C0] [ 15.924069][ C0] stack backtrace: [ 15.924073][ C0] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 7.2.0-virtme #1 PREEMPT(full) [ 15.924076][ C0] Tainted: [W]=WARN [ 15.924077][ C0] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 15.924078][ C0] Call Trace: [ 15.924080][ C0] [ 15.924081][ C0] dump_stack_lvl+0x6f/0xa0 [ 15.924085][ C0] print_irq_inversion_bug.part.0.cold+0xe6/0x143 [ 15.924087][ C0] mark_lock_irq+0x989/0x9c0 [ 15.924089][ C0] ? add_lock_to_list+0x2c/0x1a0 [ 15.924092][ C0] mark_lock+0x1d7/0xa00 [ 15.924093][ C0] mark_usage+0x42/0x170 [ 15.924095][ C0] __lock_acquire+0x388/0xc20 [ 15.924097][ C0] lock_acquire.part.0+0xd4/0x280 [ 15.924099][ C0] ? console_lock_spinning_enable+0x40/0x60 [ 15.924101][ C0] ? rcu_is_watching+0x16/0xd0 [ 15.924104][ C0] ? lock_acquire+0x13c/0x160 [ 15.924106][ C0] console_lock_spinning_enable+0x5c/0x60 [ 15.924108][ C0] ? console_lock_spinning_enable+0x40/0x60 [ 15.924109][ C0] console_emit_next_record+0x1d1/0x4c0 [ 15.924111][ C0] ? devkmsg_read+0x4b0/0x4b0 [ 15.924113][ C0] ? console_flush_one_record+0x106/0x710 [ 15.924115][ C0] ? rcu_is_watching+0x16/0xd0 [ 15.924117][ C0] ? lock_acquire+0x13c/0x160 [ 15.924119][ C0] console_flush_one_record+0x46f/0x710 [ 15.924121][ C0] ? console_emit_next_record+0x4c0/0x4c0 [ 15.924122][ C0] ? __lock_acquire+0x518/0xc20 [ 15.924125][ C0] console_unlock+0xee/0x1f0 [ 15.924126][ C0] ? console_flush_one_record+0x710/0x710 [ 15.924128][ C0] ? rcu_is_watching+0x16/0xd0 [ 15.924130][ C0] ? lock_acquire+0x60/0x160 [ 15.924132][ C0] ? __down_trylock_console_sem+0x5e/0xa0 [ 15.924133][ C0] ? vprintk_emit+0x320/0x3e0 [ 15.924135][ C0] vprintk_emit+0x37c/0x3e0 [ 15.924137][ C0] ? wake_up_klogd_work_func+0x90/0x90 [ 15.924140][ C0] ? clocksource_watchdog.part.0+0x450/0x540 [ 15.924141][ C0] _printk+0xc7/0x100 [ 15.924143][ C0] ? snapshot_read.cold+0x21/0x21 [ 15.924145][ C0] ? clocksource_watchdog.part.0+0x450/0x540 [ 15.924147][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 15.924153][ C0] clocksource_watchdog.part.0.cold+0x105/0x1dd [ 15.924155][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 15.924157][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 15.924158][ C0] call_timer_fn+0x160/0x4d0 [ 15.924160][ C0] ? detach_if_pending+0x1d0/0x1d0 [ 15.924162][ C0] ? debug_object_active_state+0x430/0x430 [ 15.924165][ C0] ? find_held_lock+0x2b/0x80 [ 15.924167][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 15.924169][ C0] ? rcu_is_watching+0x16/0xd0 [ 15.924171][ C0] __run_timers+0x68f/0xaa0 [ 15.924173][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 15.924175][ C0] ? __bpf_trace_itimer_expire+0x10/0x10 [ 15.924177][ C0] ? __lock_acquire+0x518/0xc20 [ 15.924179][ C0] ? __rwlock_init+0x150/0x150 [ 15.924182][ C0] run_timer_softirq+0xf0/0x160 [ 15.924183][ C0] ? __run_timers+0xaa0/0xaa0 [ 15.924185][ C0] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 15.924187][ C0] ? rcu_is_watching+0x16/0xd0 [ 15.924189][ C0] handle_softirqs+0x1d3/0x900 [ 15.924191][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 15.924192][ C0] ? _local_bh_enable+0xc0/0xc0 [ 15.924194][ C0] __irq_exit_rcu+0x145/0x1c0 [ 15.924196][ C0] irq_exit_rcu+0xe/0x30 [ 15.924197][ C0] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 15.924198][ C0] [ 15.924199][ C0] [ 15.924200][ C0] ? lockdep_hardirqs_on_prepare.part.0+0x9a/0x160 [ 15.924201][ C0] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 15.924203][ C0] RIP: 0010:pv_native_safe_halt+0xf/0x10 [ 15.924205][ C0] Code: 48 8b 3d 94 12 06 02 e8 1f 00 00 00 48 2b 05 58 13 a0 00 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa eb 07 0f 00 2d 13 b6 17 00 fb f4 0f 1f 40 d6 48 83 ec 20 8b 17 49 89 f8 83 e2 fe 41 89 d2 0f 01 [ 15.924207][ C0] RSP: 0018:ffffffff95a07cf8 EFLAGS: 00000296 [ 15.924209][ C0] RAX: 0000000000057489 RBX: ffffffff95a2c600 RCX: ffffffff928fa307 [ 15.924210][ C0] RDX: ffffffff95a2c600 RSI: ffffffff95878e7e RDI: ffffffff9528dce0 [ 15.924211][ C0] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 15.924212][ C0] R10: 0000000000000000 R11: 0000000000000001 R12: 1ffffffff2b40fa2 [ 15.924213][ C0] R13: 0000000000000000 R14: dffffc0000000000 R15: 0000000000014770 [ 15.924214][ C0] ? cpuidle_idle_call.constprop.0+0x237/0x410 [ 15.924216][ C0] ? lockdep_hardirqs_on+0x91/0x130 [ 15.924218][ C0] default_idle+0x9/0x10 [ 15.924219][ C0] default_idle_call+0x6e/0xb0 [ 15.924220][ C0] cpuidle_idle_call.constprop.0+0x237/0x410 [ 15.924222][ C0] ? arch_cpu_idle_exit+0x40/0x40 [ 15.924223][ C0] ? mark_tsc_async_resets+0x30/0x30 [ 15.924226][ C0] ? rcu_is_watching+0x16/0xd0 [ 15.924228][ C0] do_idle+0xd8/0x190 [ 15.924229][ C0] cpu_startup_entry+0x53/0x70 [ 15.924230][ C0] rest_init+0x279/0x280 [ 15.924232][ C0] ? __cpuidle_text_end+0x3b/0x3b [ 15.924234][ C0] ? rest_init+0x280/0x280 [ 15.924236][ C0] ? acpi_hw_set_mode+0x4a0/0x4a0 [ 15.924239][ C0] ? cpus_read_unlock+0x4f/0xc0 [ 15.924241][ C0] ? acpi_enable+0x1e4/0x330 [ 15.924244][ C0] start_kernel+0x3af/0x3b0 [ 15.924246][ C0] x86_64_start_reservations+0x24/0x30 [ 15.924248][ C0] x86_64_start_kernel+0x12b/0x130 [ 15.924250][ C0] common_startup_64+0x13e/0x148 [ 15.924253][ C0] [ 15.924257][ C0] BUG: sleeping function called from invalid context at ./include/linux/sched/mm.h:322 [ 15.924258][ C0] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 0, name: swapper/0 [ 15.924260][ C0] preempt_count: 103, expected: 0 [ 15.924260][ C0] RCU nest depth: 0, expected: 0 [ 15.924261][ C0] INFO: lockdep is turned off. [ 15.924262][ C0] irq event stamp: 357531 [ 15.924262][ C0] hardirqs last enabled at (357530): [] __down_trylock_console_sem+0x86/0xa0 [ 15.924265][ C0] hardirqs last disabled at (357531): [] console_emit_next_record+0x3d4/0x4c0 [ 15.924266][ C0] softirqs last enabled at (357470): [] handle_softirqs+0x67c/0x900 [ 15.924268][ C0] softirqs last disabled at (357515): [] __irq_exit_rcu+0x145/0x1c0 [ 15.924269][ C0] Preemption disabled at: [ 15.924270][ C0] [] schedule_preempt_disabled+0x22/0x30 [ 15.924273][ C0] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 7.2.0-virtme #1 PREEMPT(full) [ 15.924275][ C0] Tainted: [W]=WARN [ 15.924275][ C0] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 15.924276][ C0] Call Trace: [ 15.924277][ C0] [ 15.924277][ C0] dump_stack_lvl+0x6f/0xa0 [ 15.924279][ C0] ? schedule_preempt_disabled+0x22/0x30 [ 15.924281][ C0] __might_resched.cold+0x1fe/0x2c1 [ 15.924283][ C0] ? perf_trace_sched_switch+0x7c0/0x7c0 [ 15.924286][ C0] ? rcu_is_watching+0x16/0xd0 [ 15.924289][ C0] __kmalloc_noprof+0x443/0x760 [ 15.924290][ C0] ? __rwlock_init+0x150/0x150 [ 15.924292][ C0] ? alloc_buf.isra.0+0x4b/0x260 [ 15.924294][ C0] ? do_raw_spin_unlock+0x59/0x250 [ 15.924295][ C0] alloc_buf.isra.0+0x4b/0x260 [ 15.924297][ C0] put_chars+0x1e1/0x2f0 [ 15.924299][ C0] ? __send_to_port+0x420/0x420 [ 15.924301][ C0] ? mark_lock+0x1d7/0xa00 [ 15.924303][ C0] hvc_console_print+0x292/0x780 [ 15.924306][ C0] ? hvc_write+0x3a0/0x3a0 [ 15.924307][ C0] ? rcu_is_watching+0x16/0xd0 [ 15.924309][ C0] ? rcu_is_watching+0x16/0xd0 [ 15.924311][ C0] console_emit_next_record+0x22f/0x4c0 [ 15.924313][ C0] ? devkmsg_read+0x4b0/0x4b0 [ 15.924315][ C0] ? console_flush_one_record+0x106/0x710 [ 15.924317][ C0] ? rcu_is_watching+0x16/0xd0 [ 15.924319][ C0] ? lock_acquire+0x13c/0x160 [ 15.924321][ C0] console_flush_one_record+0x46f/0x710 [ 15.924323][ C0] ? console_emit_next_record+0x4c0/0x4c0 [ 15.924325][ C0] ? __lock_acquire+0x518/0xc20 [ 15.924327][ C0] console_unlock+0xee/0x1f0 [ 15.924329][ C0] ? console_flush_one_record+0x710/0x710 [ 15.924330][ C0] ? rcu_is_watching+0x16/0xd0 [ 15.924332][ C0] ? lock_acquire+0x60/0x160 [ 15.924334][ C0] ? __down_trylock_console_sem+0x5e/0xa0 [ 15.924336][ C0] ? vprintk_emit+0x320/0x3e0 [ 15.924337][ C0] vprintk_emit+0x37c/0x3e0 [ 15.924340][ C0] ? wake_up_klogd_work_func+0x90/0x90 [ 15.924342][ C0] ? clocksource_watchdog.part.0+0x450/0x540 [ 15.924344][ C0] _printk+0xc7/0x100 [ 15.924345][ C0] ? snapshot_read.cold+0x21/0x21 [ 15.924347][ C0] ? clocksource_watchdog.part.0+0x450/0x540 [ 15.924349][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 15.924351][ C0] clocksource_watchdog.part.0.cold+0x105/0x1dd [ 15.924352][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 15.924354][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 15.924356][ C0] call_timer_fn+0x160/0x4d0 [ 15.924358][ C0] ? detach_if_pending+0x1d0/0x1d0 [ 15.924359][ C0] ? debug_object_active_state+0x430/0x430 [ 15.924361][ C0] ? find_held_lock+0x2b/0x80 [ 15.924362][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 15.924364][ C0] ? rcu_is_watching+0x16/0xd0 [ 15.924366][ C0] __run_timers+0x68f/0xaa0 [ 15.924368][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 15.924370][ C0] ? __bpf_trace_itimer_expire+0x10/0x10 [ 15.924372][ C0] ? __lock_acquire+0x518/0xc20 [ 15.924375][ C0] ? __rwlock_init+0x150/0x150 [ 15.924377][ C0] run_timer_softirq+0xf0/0x160 [ 15.924379][ C0] ? __run_timers+0xaa0/0xaa0 [ 15.924380][ C0] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 15.924382][ C0] ? rcu_is_watching+0x16/0xd0 [ 15.924383][ C0] handle_softirqs+0x1d3/0x900 [ 15.924385][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 15.924387][ C0] ? _local_bh_enable+0xc0/0xc0 [ 15.924389][ C0] __irq_exit_rcu+0x145/0x1c0 [ 15.924390][ C0] irq_exit_rcu+0xe/0x30 [ 15.924391][ C0] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 15.924393][ C0] [ 15.924393][ C0] [ 15.924394][ C0] ? lockdep_hardirqs_on_prepare.part.0+0x9a/0x160 [ 15.924395][ C0] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 15.924397][ C0] RIP: 0010:pv_native_safe_halt+0xf/0x10 [ 15.924398][ C0] Code: 48 8b 3d 94 12 06 02 e8 1f 00 00 00 48 2b 05 58 13 a0 00 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa eb 07 0f 00 2d 13 b6 17 00 fb f4 0f 1f 40 d6 48 83 ec 20 8b 17 49 89 f8 83 e2 fe 41 89 d2 0f 01 [ 15.924400][ C0] RSP: 0018:ffffffff95a07cf8 EFLAGS: 00000296 [ 15.924401][ C0] RAX: 0000000000057489 RBX: ffffffff95a2c600 RCX: ffffffff928fa307 [ 15.924402][ C0] RDX: ffffffff95a2c600 RSI: ffffffff95878e7e RDI: ffffffff9528dce0 [ 15.924402][ C0] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 15.924403][ C0] R10: 0000000000000000 R11: 0000000000000001 R12: 1ffffffff2b40fa2 [ 15.924404][ C0] R13: 0000000000000000 R14: dffffc0000000000 R15: 0000000000014770 [ 15.924405][ C0] ? cpuidle_idle_call.constprop.0+0x237/0x410 [ 15.924407][ C0] ? lockdep_hardirqs_on+0x91/0x130 [ 15.924408][ C0] default_idle+0x9/0x10 [ 15.924410][ C0] default_idle_call+0x6e/0xb0 [ 15.924411][ C0] cpuidle_idle_call.constprop.0+0x237/0x410 [ 15.924413][ C0] ? arch_cpu_idle_exit+0x40/0x40 [ 15.924414][ C0] ? mark_tsc_async_resets+0x30/0x30 [ 15.924416][ C0] ? rcu_is_watching+0x16/0xd0 [ 15.924418][ C0] do_idle+0xd8/0x190 [ 15.924419][ C0] cpu_startup_entry+0x53/0x70 [ 15.924421][ C0] rest_init+0x279/0x280 [ 15.924422][ C0] ? __cpuidle_text_end+0x3b/0x3b [ 15.924425][ C0] ? rest_init+0x280/0x280 [ 15.924426][ C0] ? acpi_hw_set_mode+0x4a0/0x4a0 [ 15.924428][ C0] ? cpus_read_unlock+0x4f/0xc0 [ 15.924430][ C0] ? acpi_enable+0x1e4/0x330 [ 15.924431][ C0] start_kernel+0x3af/0x3b0 [ 15.924434][ C0] x86_64_start_reservations+0x24/0x30 [ 15.924435][ C0] x86_64_start_kernel+0x12b/0x130 [ 15.924437][ C0] common_startup_64+0x13e/0x148 [ 15.924440][ C0] [ 27.087698][ T12] RDS: shutdown called in state 1 [ 27.087713][ T12] BUG: sleeping function called from invalid context at ./include/linux/sched/mm.h:322 [ 27.087716][ T12] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 12, name: kworker/u16:0 [ 27.087718][ T12] preempt_count: 1, expected: 0 [ 27.087718][ T12] RCU nest depth: 0, expected: 0 [ 27.087719][ T12] INFO: lockdep is turned off. [ 27.087721][ T12] irq event stamp: 7362 [ 27.087721][ T12] hardirqs last enabled at (7361): [] _raw_spin_unlock_irq+0x28/0x50 [ 27.087730][ T12] hardirqs last disabled at (7362): [] __schedule+0x1246/0x20e0 [ 27.087733][ T12] softirqs last enabled at (7358): [] rds_tcp_recv_path+0xb7/0xf0 [ 27.087736][ T12] softirqs last disabled at (7356): [] release_sock+0x21/0x240 [ 27.087740][ T12] Preemption disabled at: [ 27.087741][ T12] [] vprintk_emit+0x31b/0x3e0 [ 27.087749][ T12] CPU: 3 UID: 0 PID: 12 Comm: kworker/u16:0 Tainted: G W 7.2.0-virtme #1 PREEMPT(full) [ 27.087752][ T12] Tainted: [W]=WARN [ 27.087753][ T12] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 27.087755][ T12] Workqueue: krds_cp_wq#1/6 rds_shutdown_worker [ 27.087760][ T12] Call Trace: [ 27.087761][ T12] [ 27.087763][ T12] dump_stack_lvl+0x6f/0xa0 [ 27.087768][ T12] ? vprintk_emit+0x31b/0x3e0 [ 27.087770][ T12] __might_resched.cold+0x1fe/0x2c1 [ 27.087773][ T12] ? perf_trace_sched_switch+0x7c0/0x7c0 [ 27.087776][ T12] ? prb_next_reserve_seq+0x2d0/0x2d0 [ 27.087778][ T12] ? rcu_is_watching+0x16/0xd0 [ 27.087782][ T12] __kmalloc_noprof+0x443/0x760 [ 27.087786][ T12] ? __rwlock_init+0x150/0x150 [ 27.087789][ T12] ? alloc_buf.isra.0+0x4b/0x260 [ 27.087792][ T12] ? do_raw_spin_unlock+0x59/0x250 [ 27.087794][ T12] alloc_buf.isra.0+0x4b/0x260 [ 27.087796][ T12] put_chars+0x1e1/0x2f0 [ 27.087798][ T12] ? prb_final_commit+0x50/0x50 [ 27.087799][ T12] ? __send_to_port+0x420/0x420 [ 27.087801][ T12] ? printk_get_next_message+0x240/0x650 [ 27.087804][ T12] hvc_console_print+0x292/0x780 [ 27.087807][ T12] ? hvc_write+0x3a0/0x3a0 [ 27.087809][ T12] ? do_raw_spin_lock+0x131/0x280 [ 27.087810][ T12] ? vsnprintf+0x668/0x1700 [ 27.087812][ T12] ? __rwlock_init+0x150/0x150 [ 27.087814][ T12] ? rcu_is_watching+0x16/0xd0 [ 27.087816][ T12] ? lock_acquire+0x13c/0x160 [ 27.087818][ T12] console_emit_next_record+0x22f/0x4c0 [ 27.087820][ T12] ? devkmsg_read+0x4b0/0x4b0 [ 27.087822][ T12] ? rcu_is_watching+0x16/0xd0 [ 27.087824][ T12] ? rcu_is_watching+0x16/0xd0 [ 27.087825][ T12] ? rcu_is_watching+0x16/0xd0 [ 27.087827][ T12] ? lock_acquire+0x13c/0x160 [ 27.087829][ T12] ? console_flush_one_record+0x111/0x710 [ 27.087830][ T12] console_flush_one_record+0x46f/0x710 [ 27.087833][ T12] ? console_emit_next_record+0x4c0/0x4c0 [ 27.087835][ T12] console_unlock+0xee/0x1f0 [ 27.087837][ T12] ? lock_acquire+0x13c/0x160 [ 27.087839][ T12] ? console_flush_one_record+0x710/0x710 [ 27.087840][ T12] ? rcu_is_watching+0x16/0xd0 [ 27.087842][ T12] ? lock_release+0x184/0x1f0 [ 27.087843][ T12] ? lock_acquire+0x60/0x160 [ 27.087845][ T12] ? __rcu_read_lock+0x30/0x70 [ 27.087847][ T12] ? __down_trylock_console_sem+0x5e/0xa0 [ 27.087848][ T12] ? vprintk_emit+0x320/0x3e0 [ 27.087850][ T12] vprintk_emit+0x37c/0x3e0 [ 27.087852][ T12] ? wake_up_klogd_work_func+0x90/0x90 [ 27.087854][ T12] ? rcu_is_watching+0x16/0xd0 [ 27.087856][ T12] ? ww_mutex_lock+0x160/0x160 [ 27.087858][ T12] ? try_to_grab_pending+0x77/0x840 [ 27.087861][ T12] __rds_conn_path_error+0xbd/0x100 [ 27.087864][ T12] ? rds_check_all_paths+0xb0/0xb0 [ 27.087865][ T12] ? lock_acquire+0x13c/0x160 [ 27.087867][ T12] ? rcu_is_watching+0x16/0xd0 [ 27.087869][ T12] ? rds_conn_shutdown+0x652/0xa20 [ 27.087870][ T12] rds_conn_shutdown+0x695/0xa20 [ 27.087872][ T12] ? __rds_conn_path_error+0x100/0x100 [ 27.087873][ T12] ? debug_object_active_state+0x430/0x430 [ 27.087877][ T12] ? rcu_is_watching+0x16/0xd0 [ 27.087879][ T12] ? rcu_is_watching+0x16/0xd0 [ 27.087880][ T12] ? lock_acquire+0x13c/0x160 [ 27.087882][ T12] ? rcu_is_watching+0x16/0xd0 [ 27.087884][ T12] ? process_one_work+0xdfc/0x1560 [ 27.087885][ T12] process_one_work+0xe3e/0x1560 [ 27.087888][ T12] ? __queue_delayed_work+0x470/0x470 [ 27.087890][ T12] ? kthread_data+0x7e/0xc0 [ 27.087893][ T12] ? __rwlock_init+0x150/0x150 [ 27.087895][ T12] worker_thread+0x4f1/0xd60 [ 27.087897][ T12] ? rescuer_thread+0x1340/0x1340 [ 27.087899][ T12] kthread+0x367/0x460 [ 27.087901][ T12] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 27.087903][ T12] ? kthread_affine_preferred+0x4c0/0x4c0 [ 27.087905][ T12] ret_from_fork+0x474/0x6b0 [ 27.087908][ T12] ? arch_exit_to_user_mode_prepare.isra.0+0x120/0x120 [ 27.087909][ T12] ? __switch_to+0x5a3/0xe00 [ 27.087912][ T12] ? kthread_affine_preferred+0x4c0/0x4c0 [ 27.087914][ T12] ret_from_fork_asm+0x11/0x20 [ 27.087917][ T12] [ 34.317262][ T69] RDS: shutdown called in state 1 [ 34.317325][ T69] BUG: sleeping function called from invalid context at ./include/linux/sched/mm.h:322 [ 34.317328][ T69] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 69, name: kworker/u16:1 [ 34.317330][ T69] preempt_count: 1, expected: 0 [ 34.317331][ T69] RCU nest depth: 0, expected: 0 [ 34.317332][ T69] INFO: lockdep is turned off. [ 34.317333][ T69] irq event stamp: 14976 [ 34.317334][ T69] hardirqs last enabled at (14975): [] _raw_spin_unlock_irq+0x28/0x50 [ 34.317342][ T69] hardirqs last disabled at (14976): [] __schedule+0x1246/0x20e0 [ 34.317345][ T69] softirqs last enabled at (14972): [] rds_tcp_recv_path+0xb7/0xf0 [ 34.317348][ T69] softirqs last disabled at (14970): [] release_sock+0x21/0x240 [ 34.317352][ T69] Preemption disabled at: [ 34.317352][ T69] [] vprintk_emit+0x31b/0x3e0 [ 34.317360][ T69] CPU: 1 UID: 0 PID: 69 Comm: kworker/u16:1 Tainted: G W 7.2.0-virtme #1 PREEMPT(full) [ 34.317364][ T69] Tainted: [W]=WARN [ 34.317365][ T69] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 34.317367][ T69] Workqueue: krds_cp_wq#1/3 rds_shutdown_worker [ 34.317371][ T69] Call Trace: [ 34.317373][ T69] [ 34.317375][ T69] dump_stack_lvl+0x6f/0xa0 [ 34.317380][ T69] ? vprintk_emit+0x31b/0x3e0 [ 34.317382][ T69] __might_resched.cold+0x1fe/0x2c1 [ 34.317386][ T69] ? perf_trace_sched_switch+0x7c0/0x7c0 [ 34.317389][ T69] ? prb_next_reserve_seq+0x2d0/0x2d0 [ 34.317390][ T69] ? rcu_is_watching+0x16/0xd0 [ 34.317394][ T69] __kmalloc_noprof+0x443/0x760 [ 34.317397][ T69] ? __rwlock_init+0x150/0x150 [ 34.317400][ T69] ? alloc_buf.isra.0+0x4b/0x260 [ 34.317403][ T69] ? do_raw_spin_unlock+0x59/0x250 [ 34.317405][ T69] alloc_buf.isra.0+0x4b/0x260 [ 34.317407][ T69] put_chars+0x1e1/0x2f0 [ 34.317409][ T69] ? prb_final_commit+0x50/0x50 [ 34.317410][ T69] ? __send_to_port+0x420/0x420 [ 34.317412][ T69] ? printk_get_next_message+0x240/0x650 [ 34.317414][ T69] hvc_console_print+0x292/0x780 [ 34.317418][ T69] ? hvc_write+0x3a0/0x3a0 [ 34.317419][ T69] ? do_raw_spin_lock+0x131/0x280 [ 34.317421][ T69] ? vsnprintf+0x668/0x1700 [ 34.317423][ T69] ? __rwlock_init+0x150/0x150 [ 34.317424][ T69] ? rcu_is_watching+0x16/0xd0 [ 34.317426][ T69] ? lock_acquire+0x13c/0x160 [ 34.317428][ T69] console_emit_next_record+0x22f/0x4c0 [ 34.317431][ T69] ? devkmsg_read+0x4b0/0x4b0 [ 34.317432][ T69] ? rcu_is_watching+0x16/0xd0 [ 34.317434][ T69] ? rcu_is_watching+0x16/0xd0 [ 34.317436][ T69] ? rcu_is_watching+0x16/0xd0 [ 34.317437][ T69] ? lock_acquire+0x13c/0x160 [ 34.317439][ T69] ? console_flush_one_record+0x111/0x710 [ 34.317441][ T69] console_flush_one_record+0x46f/0x710 [ 34.317443][ T69] ? console_emit_next_record+0x4c0/0x4c0 [ 34.317445][ T69] console_unlock+0xee/0x1f0 [ 34.317447][ T69] ? lock_acquire+0x13c/0x160 [ 34.317448][ T69] ? console_flush_one_record+0x710/0x710 [ 34.317450][ T69] ? rcu_is_watching+0x16/0xd0 [ 34.317451][ T69] ? lock_release+0x184/0x1f0 [ 34.317453][ T69] ? lock_acquire+0x60/0x160 [ 34.317454][ T69] ? __rcu_read_lock+0x30/0x70 [ 34.317456][ T69] ? __down_trylock_console_sem+0x5e/0xa0 [ 34.317458][ T69] ? vprintk_emit+0x320/0x3e0 [ 34.317460][ T69] vprintk_emit+0x37c/0x3e0 [ 34.317462][ T69] ? wake_up_klogd_work_func+0x90/0x90 [ 34.317464][ T69] ? lock_release+0x184/0x1f0 [ 34.317465][ T69] ? ww_mutex_lock+0x160/0x160 [ 34.317467][ T69] ? __queue_work+0x3ab/0xd30 [ 34.317470][ T69] __rds_conn_path_error+0xbd/0x100 [ 34.317473][ T69] ? rds_check_all_paths+0xb0/0xb0 [ 34.317474][ T69] ? lock_acquire+0x13c/0x160 [ 34.317476][ T69] ? rcu_is_watching+0x16/0xd0 [ 34.317478][ T69] ? rds_conn_shutdown+0x652/0xa20 [ 34.317479][ T69] rds_conn_shutdown+0x695/0xa20 [ 34.317481][ T69] ? __rds_conn_path_error+0x100/0x100 [ 34.317482][ T69] ? debug_object_active_state+0x430/0x430 [ 34.317486][ T69] ? rcu_is_watching+0x16/0xd0 [ 34.317488][ T69] ? rcu_is_watching+0x16/0xd0 [ 34.317489][ T69] ? lock_acquire+0x13c/0x160 [ 34.317491][ T69] ? rcu_is_watching+0x16/0xd0 [ 34.317493][ T69] ? process_one_work+0xdfc/0x1560 [ 34.317495][ T69] process_one_work+0xe3e/0x1560 [ 34.317497][ T69] ? __queue_delayed_work+0x470/0x470 [ 34.317500][ T69] ? __rwlock_init+0x150/0x150 [ 34.317502][ T69] worker_thread+0x4f1/0xd60 [ 34.317505][ T69] ? rescuer_thread+0x1340/0x1340 [ 34.317506][ T69] kthread+0x367/0x460 [ 34.317509][ T69] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 34.317511][ T69] ? kthread_affine_preferred+0x4c0/0x4c0 [ 34.317513][ T69] ret_from_fork+0x474/0x6b0 [ 34.317516][ T69] ? arch_exit_to_user_mode_prepare.isra.0+0x120/0x120 [ 34.317517][ T69] ? __switch_to+0x5a3/0xe00 [ 34.317520][ T69] ? kthread_affine_preferred+0x4c0/0x4c0 [ 34.317522][ T69] ret_from_fork_asm+0x11/0x20 [ 34.317525][ T69]