[ 13.946828][ C0] clocksource: Watchdog remote CPU 2 read timed out [ 13.947020][ C0] [ 13.947022][ C0] ======================================================== [ 13.947023][ C0] WARNING: possible irq lock inversion dependency detected [ 13.947025][ C0] 7.2.0-virtme #1 Tainted: G W [ 13.947027][ C0] -------------------------------------------------------- [ 13.947027][ C0] swapper/0/0 just changed the state of lock: [ 13.947029][ C0] ffffffffa3699660 (console_owner){..-.}-{0:0}, at: console_lock_spinning_enable+0x40/0x60 [ 13.947041][ C0] but this lock took another, SOFTIRQ-unsafe lock in the past: [ 13.947043][ C0] (fs_reclaim){+.+.}-{0:0} [ 13.947044][ C0] [ 13.947044][ C0] [ 13.947044][ C0] and interrupts could create inverse lock ordering between them. [ 13.947044][ C0] [ 13.947045][ C0] [ 13.947045][ C0] other info that might help us debug this: [ 13.947046][ C0] Possible interrupt unsafe locking scenario: [ 13.947046][ C0] [ 13.947046][ C0] CPU0 CPU1 [ 13.947047][ C0] ---- ---- [ 13.947047][ C0] lock(fs_reclaim); [ 13.947048][ C0] local_irq_disable(); [ 13.947048][ C0] lock(console_owner); [ 13.947049][ C0] lock(fs_reclaim); [ 13.947050][ C0] [ 13.947051][ C0] lock(console_owner); [ 13.947051][ C0] [ 13.947051][ C0] *** DEADLOCK *** [ 13.947051][ C0] [ 13.947052][ C0] locks held by swapper/0/0: 4, last CPU#0: [ 13.947053][ C0] #0: ffa0000000007c90 ((&watchdog_timer)){+.-.}-{0:0}, at: call_timer_fn+0x110/0x4d0 [ 13.947058][ C0] #1: ffffffffa37de8b8 (watchdog_lock){+.-.}-{3:3}, at: clocksource_watchdog+0x15/0x30 [ 13.947062][ C0] #2: ffffffffa3779cc0 (console_lock){+.+.}-{0:0}, at: vprintk_emit+0x320/0x3e0 [ 13.947066][ C0] #3: ffffffffa3779d38 (console_srcu){....}-{0:0}, at: console_flush_one_record+0x106/0x710 [ 13.947069][ C0] [ 13.947069][ C0] the shortest dependencies between 2nd lock and 1st lock: [ 13.947073][ C0] -> (fs_reclaim){+.+.}-{0:0} { [ 13.947075][ C0] HARDIRQ-ON-W at: [ 13.947077][ C0] __lock_acquire+0x388/0xc20 [ 13.947080][ C0] lock_acquire.part.0+0xd4/0x280 [ 13.947081][ C0] fs_reclaim_acquire+0xd5/0x120 [ 13.947084][ C0] __kmalloc_cache_noprof+0x6e/0x620 [ 13.947086][ C0] kthread_create_worker_on_node+0xea/0x210 [ 13.947089][ C0] workqueue_init+0x2a/0x680 [ 13.947093][ C0] kernel_init_freeable+0x2fe/0x630 [ 13.947095][ C0] kernel_init+0x21/0x150 [ 13.947099][ C0] ret_from_fork+0x474/0x6b0 [ 13.947102][ C0] ret_from_fork_asm+0x11/0x20 [ 13.947105][ C0] SOFTIRQ-ON-W at: [ 13.947105][ C0] __lock_acquire+0x388/0xc20 [ 13.947107][ C0] lock_acquire.part.0+0xd4/0x280 [ 13.947108][ C0] fs_reclaim_acquire+0xd5/0x120 [ 13.947109][ C0] __kmalloc_cache_noprof+0x6e/0x620 [ 13.947111][ C0] kthread_create_worker_on_node+0xea/0x210 [ 13.947112][ C0] workqueue_init+0x2a/0x680 [ 13.947113][ C0] kernel_init_freeable+0x2fe/0x630 [ 13.947115][ C0] kernel_init+0x21/0x150 [ 13.947116][ C0] ret_from_fork+0x474/0x6b0 [ 13.947118][ C0] ret_from_fork_asm+0x11/0x20 [ 13.947119][ C0] INITIAL USE at: [ 13.947120][ C0] __lock_acquire+0x388/0xc20 [ 13.947121][ C0] lock_acquire.part.0+0xd4/0x280 [ 13.947122][ C0] fs_reclaim_acquire+0xd5/0x120 [ 13.947124][ C0] __kmalloc_cache_noprof+0x6e/0x620 [ 13.947125][ C0] kthread_create_worker_on_node+0xea/0x210 [ 13.947126][ C0] workqueue_init+0x2a/0x680 [ 13.947127][ C0] kernel_init_freeable+0x2fe/0x630 [ 13.947129][ C0] kernel_init+0x21/0x150 [ 13.947130][ C0] ret_from_fork+0x474/0x6b0 [ 13.947132][ C0] ret_from_fork_asm+0x11/0x20 [ 13.947133][ C0] } [ 13.947133][ C0] ... key at: [] __fs_reclaim_map+0x0/0xbe0 [ 13.947138][ C0] ... acquired at: [ 13.947139][ C0] __lock_acquire+0x518/0xc20 [ 13.947140][ C0] lock_acquire.part.0+0xd4/0x280 [ 13.947141][ C0] fs_reclaim_acquire+0xd5/0x120 [ 13.947142][ C0] __kmalloc_noprof+0xd3/0x760 [ 13.947144][ C0] alloc_buf.isra.0+0x4b/0x260 [ 13.947147][ C0] put_chars+0x1e1/0x2f0 [ 13.947148][ C0] hvc_console_print+0x292/0x780 [ 13.947151][ C0] console_emit_next_record+0x22f/0x4c0 [ 13.947152][ C0] console_flush_one_record+0x46f/0x710 [ 13.947154][ C0] console_unlock+0xee/0x1f0 [ 13.947156][ C0] vprintk_emit+0x37c/0x3e0 [ 13.947157][ C0] _printk+0xc7/0x100 [ 13.947160][ C0] print_ipi_mode+0x2e/0x40 [ 13.947162][ C0] do_one_initcall+0x124/0x4f0 [ 13.947164][ C0] kernel_init_freeable+0x596/0x630 [ 13.947165][ C0] kernel_init+0x21/0x150 [ 13.947167][ C0] ret_from_fork+0x474/0x6b0 [ 13.947168][ C0] ret_from_fork_asm+0x11/0x20 [ 13.947169][ C0] [ 13.947169][ C0] -> (console_owner){..-.}-{0:0} { [ 13.947171][ C0] IN-SOFTIRQ-W at: [ 13.947172][ C0] __lock_acquire+0x388/0xc20 [ 13.947173][ C0] lock_acquire.part.0+0xd4/0x280 [ 13.947175][ C0] console_lock_spinning_enable+0x5c/0x60 [ 13.947176][ C0] console_emit_next_record+0x1d1/0x4c0 [ 13.947178][ C0] console_flush_one_record+0x46f/0x710 [ 13.947180][ C0] console_unlock+0xee/0x1f0 [ 13.947181][ C0] vprintk_emit+0x37c/0x3e0 [ 13.947183][ C0] _printk+0xc7/0x100 [ 13.947184][ C0] clocksource_watchdog.part.0.cold+0x105/0x1dd [ 13.947187][ C0] call_timer_fn+0x160/0x4d0 [ 13.947188][ C0] __run_timers+0x68f/0xaa0 [ 13.947190][ C0] run_timer_softirq+0xf0/0x160 [ 13.947191][ C0] handle_softirqs+0x1d3/0x900 [ 13.947194][ C0] __irq_exit_rcu+0x145/0x1c0 [ 13.947195][ C0] irq_exit_rcu+0xe/0x30 [ 13.947197][ C0] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 13.947198][ C0] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 13.947200][ C0] pv_native_safe_halt+0xf/0x10 [ 13.947202][ C0] default_idle+0x9/0x10 [ 13.947203][ C0] default_idle_call+0x6e/0xb0 [ 13.947204][ C0] cpuidle_idle_call.constprop.0+0x237/0x410 [ 13.947206][ C0] do_idle+0xd8/0x190 [ 13.947208][ C0] cpu_startup_entry+0x53/0x70 [ 13.947209][ C0] rest_init+0x279/0x280 [ 13.947210][ C0] start_kernel+0x3af/0x3b0 [ 13.947212][ C0] x86_64_start_reservations+0x24/0x30 [ 13.947214][ C0] x86_64_start_kernel+0x12b/0x130 [ 13.947215][ C0] common_startup_64+0x13e/0x148 [ 13.947218][ C0] INITIAL USE at: [ 13.947219][ C0] } [ 13.947219][ C0] ... key at: [] console_owner_dep_map+0x0/0x60 [ 13.947223][ C0] ... acquired at: [ 13.947223][ C0] mark_lock+0x1d7/0xa00 [ 13.947225][ C0] mark_usage+0x42/0x170 [ 13.947226][ C0] __lock_acquire+0x388/0xc20 [ 13.947228][ C0] lock_acquire.part.0+0xd4/0x280 [ 13.947229][ C0] console_lock_spinning_enable+0x5c/0x60 [ 13.947230][ C0] console_emit_next_record+0x1d1/0x4c0 [ 13.947232][ C0] console_flush_one_record+0x46f/0x710 [ 13.947234][ C0] console_unlock+0xee/0x1f0 [ 13.947235][ C0] vprintk_emit+0x37c/0x3e0 [ 13.947237][ C0] _printk+0xc7/0x100 [ 13.947238][ C0] clocksource_watchdog.part.0.cold+0x105/0x1dd [ 13.947239][ C0] call_timer_fn+0x160/0x4d0 [ 13.947241][ C0] __run_timers+0x68f/0xaa0 [ 13.947242][ C0] run_timer_softirq+0xf0/0x160 [ 13.947244][ C0] handle_softirqs+0x1d3/0x900 [ 13.947245][ C0] __irq_exit_rcu+0x145/0x1c0 [ 13.947246][ C0] irq_exit_rcu+0xe/0x30 [ 13.947248][ C0] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 13.947249][ C0] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 13.947250][ C0] pv_native_safe_halt+0xf/0x10 [ 13.947251][ C0] default_idle+0x9/0x10 [ 13.947252][ C0] default_idle_call+0x6e/0xb0 [ 13.947253][ C0] cpuidle_idle_call.constprop.0+0x237/0x410 [ 13.947255][ C0] do_idle+0xd8/0x190 [ 13.947256][ C0] cpu_startup_entry+0x53/0x70 [ 13.947257][ C0] rest_init+0x279/0x280 [ 13.947258][ C0] start_kernel+0x3af/0x3b0 [ 13.947260][ C0] x86_64_start_reservations+0x24/0x30 [ 13.947261][ C0] x86_64_start_kernel+0x12b/0x130 [ 13.947263][ C0] common_startup_64+0x13e/0x148 [ 13.947264][ C0] [ 13.947264][ C0] [ 13.947264][ C0] stack backtrace: [ 13.947267][ C0] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 7.2.0-virtme #1 PREEMPT(full) [ 13.947271][ C0] Tainted: [W]=WARN [ 13.947272][ C0] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 13.947273][ C0] Call Trace: [ 13.947274][ C0] [ 13.947276][ C0] dump_stack_lvl+0x6f/0xa0 [ 13.947280][ C0] print_irq_inversion_bug.part.0.cold+0xe6/0x143 [ 13.947282][ C0] mark_lock_irq+0x989/0x9c0 [ 13.947284][ C0] ? add_lock_to_list+0x2c/0x1a0 [ 13.947286][ C0] mark_lock+0x1d7/0xa00 [ 13.947288][ C0] mark_usage+0x42/0x170 [ 13.947290][ C0] __lock_acquire+0x388/0xc20 [ 13.947292][ C0] lock_acquire.part.0+0xd4/0x280 [ 13.947294][ C0] ? console_lock_spinning_enable+0x40/0x60 [ 13.947296][ C0] ? rcu_is_watching+0x16/0xd0 [ 13.947299][ C0] ? lock_acquire+0x13c/0x160 [ 13.947301][ C0] console_lock_spinning_enable+0x5c/0x60 [ 13.947303][ C0] ? console_lock_spinning_enable+0x40/0x60 [ 13.947305][ C0] console_emit_next_record+0x1d1/0x4c0 [ 13.947307][ C0] ? devkmsg_read+0x4b0/0x4b0 [ 13.947309][ C0] ? console_flush_one_record+0x106/0x710 [ 13.947311][ C0] ? rcu_is_watching+0x16/0xd0 [ 13.947313][ C0] ? lock_acquire+0x13c/0x160 [ 13.947315][ C0] console_flush_one_record+0x46f/0x710 [ 13.947317][ C0] ? console_emit_next_record+0x4c0/0x4c0 [ 13.947319][ C0] ? __lock_acquire+0x518/0xc20 [ 13.947321][ C0] console_unlock+0xee/0x1f0 [ 13.947323][ C0] ? console_flush_one_record+0x710/0x710 [ 13.947324][ C0] ? rcu_is_watching+0x16/0xd0 [ 13.947326][ C0] ? lock_acquire+0x60/0x160 [ 13.947328][ C0] ? __down_trylock_console_sem+0x5e/0xa0 [ 13.947330][ C0] ? vprintk_emit+0x320/0x3e0 [ 13.947332][ C0] vprintk_emit+0x37c/0x3e0 [ 13.947334][ C0] ? wake_up_klogd_work_func+0x90/0x90 [ 13.947337][ C0] ? clocksource_watchdog.part.0+0x450/0x540 [ 13.947338][ C0] _printk+0xc7/0x100 [ 13.947340][ C0] ? snapshot_read.cold+0x21/0x21 [ 13.947342][ C0] ? clocksource_watchdog.part.0+0x450/0x540 [ 13.947344][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 13.947346][ C0] clocksource_watchdog.part.0.cold+0x105/0x1dd [ 13.947347][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 13.947349][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 13.947351][ C0] call_timer_fn+0x160/0x4d0 [ 13.947353][ C0] ? detach_if_pending+0x1d0/0x1d0 [ 13.947354][ C0] ? debug_object_active_state+0x430/0x430 [ 13.947358][ C0] ? find_held_lock+0x2b/0x80 [ 13.947360][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 13.947362][ C0] ? rcu_is_watching+0x16/0xd0 [ 13.947364][ C0] __run_timers+0x68f/0xaa0 [ 13.947366][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 13.947368][ C0] ? __bpf_trace_itimer_expire+0x10/0x10 [ 13.947370][ C0] ? __lock_acquire+0x518/0xc20 [ 13.947372][ C0] ? __rwlock_init+0x150/0x150 [ 13.947375][ C0] run_timer_softirq+0xf0/0x160 [ 13.947377][ C0] ? __run_timers+0xaa0/0xaa0 [ 13.947379][ C0] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 13.947381][ C0] ? rcu_is_watching+0x16/0xd0 [ 13.947383][ C0] handle_softirqs+0x1d3/0x900 [ 13.947384][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 13.947386][ C0] ? _local_bh_enable+0xc0/0xc0 [ 13.947388][ C0] __irq_exit_rcu+0x145/0x1c0 [ 13.947390][ C0] irq_exit_rcu+0xe/0x30 [ 13.947391][ C0] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 13.947393][ C0] [ 13.947393][ C0] [ 13.947394][ C0] ? lockdep_hardirqs_on_prepare.part.0+0x9a/0x160 [ 13.947395][ C0] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 13.947397][ C0] RIP: 0010:pv_native_safe_halt+0xf/0x10 [ 13.947400][ C0] Code: 48 8b 3d 94 02 06 02 e8 1f 00 00 00 48 2b 05 58 03 a0 00 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa eb 07 0f 00 2d 13 a6 17 00 fb f4 0f 1f 40 d6 48 83 ec 20 8b 17 49 89 f8 83 e2 fe 41 89 d2 0f 01 [ 13.947401][ C0] RSP: 0018:ffffffffa3407cf8 EFLAGS: 00000296 [ 13.947404][ C0] RAX: 00000000000611ff RBX: ffffffffa342c600 RCX: ffffffffa02fa307 [ 13.947405][ C0] RDX: ffffffffa342c600 RSI: ffffffffa3278e7e RDI: ffffffffa2c8dce0 [ 13.947406][ C0] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 13.947407][ C0] R10: 0000000000000000 R11: 0000000000000001 R12: 1ffffffff4680fa2 [ 13.947407][ C0] R13: 0000000000000000 R14: dffffc0000000000 R15: 0000000000014770 [ 13.947409][ C0] ? cpuidle_idle_call.constprop.0+0x237/0x410 [ 13.947411][ C0] ? lockdep_hardirqs_on+0x91/0x130 [ 13.947412][ C0] default_idle+0x9/0x10 [ 13.947414][ C0] default_idle_call+0x6e/0xb0 [ 13.947415][ C0] cpuidle_idle_call.constprop.0+0x237/0x410 [ 13.947417][ C0] ? arch_cpu_idle_exit+0x40/0x40 [ 13.947418][ C0] ? mark_tsc_async_resets+0x30/0x30 [ 13.947421][ C0] ? rcu_is_watching+0x16/0xd0 [ 13.947423][ C0] do_idle+0xd8/0x190 [ 13.947424][ C0] cpu_startup_entry+0x53/0x70 [ 13.947426][ C0] rest_init+0x279/0x280 [ 13.947428][ C0] ? __cpuidle_text_end+0x3b/0x3b [ 13.947430][ C0] ? rest_init+0x280/0x280 [ 13.947432][ C0] ? acpi_hw_set_mode+0x4a0/0x4a0 [ 13.947434][ C0] ? cpus_read_unlock+0x4f/0xc0 [ 13.947437][ C0] ? acpi_enable+0x1e4/0x330 [ 13.947440][ C0] start_kernel+0x3af/0x3b0 [ 13.947442][ C0] x86_64_start_reservations+0x24/0x30 [ 13.947444][ C0] x86_64_start_kernel+0x12b/0x130 [ 13.947445][ C0] common_startup_64+0x13e/0x148 [ 13.947453][ C0] [ 13.947457][ C0] BUG: sleeping function called from invalid context at ./include/linux/sched/mm.h:322 [ 13.947459][ C0] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 0, name: swapper/0 [ 13.947460][ C0] preempt_count: 103, expected: 0 [ 13.947461][ C0] RCU nest depth: 0, expected: 0 [ 13.947461][ C0] INFO: lockdep is turned off. [ 13.947462][ C0] irq event stamp: 397841 [ 13.947463][ C0] hardirqs last enabled at (397840): [] asm_sysvec_call_function_single+0x1a/0x20 [ 13.947465][ C0] hardirqs last disabled at (397841): [] console_emit_next_record+0x3d4/0x4c0 [ 13.947467][ C0] softirqs last enabled at (397800): [] handle_softirqs+0x67c/0x900 [ 13.947468][ C0] softirqs last disabled at (397825): [] __irq_exit_rcu+0x145/0x1c0 [ 13.947470][ C0] Preemption disabled at: [ 13.947470][ C0] [] schedule_preempt_disabled+0x22/0x30 [ 13.947473][ C0] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 7.2.0-virtme #1 PREEMPT(full) [ 13.947475][ C0] Tainted: [W]=WARN [ 13.947476][ C0] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 13.947476][ C0] Call Trace: [ 13.947477][ C0] [ 13.947477][ C0] dump_stack_lvl+0x6f/0xa0 [ 13.947479][ C0] ? schedule_preempt_disabled+0x22/0x30 [ 13.947481][ C0] __might_resched.cold+0x1fe/0x2c1 [ 13.947484][ C0] ? perf_trace_sched_switch+0x7c0/0x7c0 [ 13.947487][ C0] ? rcu_is_watching+0x16/0xd0 [ 13.947489][ C0] __kmalloc_noprof+0x443/0x760 [ 13.947491][ C0] ? __rwlock_init+0x150/0x150 [ 13.947492][ C0] ? alloc_buf.isra.0+0x4b/0x260 [ 13.947494][ C0] ? do_raw_spin_unlock+0x59/0x250 [ 13.947496][ C0] alloc_buf.isra.0+0x4b/0x260 [ 13.947498][ C0] put_chars+0x1e1/0x2f0 [ 13.947500][ C0] ? __send_to_port+0x420/0x420 [ 13.947502][ C0] ? mark_lock+0x1d7/0xa00 [ 13.947504][ C0] hvc_console_print+0x292/0x780 [ 13.947507][ C0] ? hvc_write+0x3a0/0x3a0 [ 13.947508][ C0] ? rcu_is_watching+0x16/0xd0 [ 13.947510][ C0] ? rcu_is_watching+0x16/0xd0 [ 13.947513][ C0] console_emit_next_record+0x22f/0x4c0 [ 13.947515][ C0] ? devkmsg_read+0x4b0/0x4b0 [ 13.947516][ C0] ? console_flush_one_record+0x106/0x710 [ 13.947518][ C0] ? rcu_is_watching+0x16/0xd0 [ 13.947520][ C0] ? lock_acquire+0x13c/0x160 [ 13.947522][ C0] console_flush_one_record+0x46f/0x710 [ 13.947524][ C0] ? console_emit_next_record+0x4c0/0x4c0 [ 13.947526][ C0] ? __lock_acquire+0x518/0xc20 [ 13.947528][ C0] console_unlock+0xee/0x1f0 [ 13.947530][ C0] ? console_flush_one_record+0x710/0x710 [ 13.947532][ C0] ? rcu_is_watching+0x16/0xd0 [ 13.947534][ C0] ? lock_acquire+0x60/0x160 [ 13.947536][ C0] ? __down_trylock_console_sem+0x5e/0xa0 [ 13.947537][ C0] ? vprintk_emit+0x320/0x3e0 [ 13.947539][ C0] vprintk_emit+0x37c/0x3e0 [ 13.947541][ C0] ? wake_up_klogd_work_func+0x90/0x90 [ 13.947544][ C0] ? clocksource_watchdog.part.0+0x450/0x540 [ 13.947545][ C0] _printk+0xc7/0x100 [ 13.947547][ C0] ? snapshot_read.cold+0x21/0x21 [ 13.947549][ C0] ? clocksource_watchdog.part.0+0x450/0x540 [ 13.947551][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 13.947553][ C0] clocksource_watchdog.part.0.cold+0x105/0x1dd [ 13.947554][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 13.947556][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 13.947558][ C0] call_timer_fn+0x160/0x4d0 [ 13.947560][ C0] ? detach_if_pending+0x1d0/0x1d0 [ 13.947562][ C0] ? debug_object_active_state+0x430/0x430 [ 13.947563][ C0] ? find_held_lock+0x2b/0x80 [ 13.947565][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 13.947567][ C0] ? rcu_is_watching+0x16/0xd0 [ 13.947569][ C0] __run_timers+0x68f/0xaa0 [ 13.947571][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 13.947573][ C0] ? __bpf_trace_itimer_expire+0x10/0x10 [ 13.947575][ C0] ? __lock_acquire+0x518/0xc20 [ 13.947577][ C0] ? __rwlock_init+0x150/0x150 [ 13.947580][ C0] run_timer_softirq+0xf0/0x160 [ 13.947582][ C0] ? __run_timers+0xaa0/0xaa0 [ 13.947583][ C0] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 13.947585][ C0] ? rcu_is_watching+0x16/0xd0 [ 13.947586][ C0] handle_softirqs+0x1d3/0x900 [ 13.947588][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 13.947590][ C0] ? _local_bh_enable+0xc0/0xc0 [ 13.947592][ C0] __irq_exit_rcu+0x145/0x1c0 [ 13.947593][ C0] irq_exit_rcu+0xe/0x30 [ 13.947595][ C0] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 13.947596][ C0] [ 13.947597][ C0] [ 13.947597][ C0] ? lockdep_hardirqs_on_prepare.part.0+0x9a/0x160 [ 13.947599][ C0] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 13.947600][ C0] RIP: 0010:pv_native_safe_halt+0xf/0x10 [ 13.947602][ C0] Code: 48 8b 3d 94 02 06 02 e8 1f 00 00 00 48 2b 05 58 03 a0 00 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa eb 07 0f 00 2d 13 a6 17 00 fb f4 0f 1f 40 d6 48 83 ec 20 8b 17 49 89 f8 83 e2 fe 41 89 d2 0f 01 [ 13.947603][ C0] RSP: 0018:ffffffffa3407cf8 EFLAGS: 00000296 [ 13.947604][ C0] RAX: 00000000000611ff RBX: ffffffffa342c600 RCX: ffffffffa02fa307 [ 13.947605][ C0] RDX: ffffffffa342c600 RSI: ffffffffa3278e7e RDI: ffffffffa2c8dce0 [ 13.947605][ C0] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 13.947606][ C0] R10: 0000000000000000 R11: 0000000000000001 R12: 1ffffffff4680fa2 [ 13.947607][ C0] R13: 0000000000000000 R14: dffffc0000000000 R15: 0000000000014770 [ 13.947608][ C0] ? cpuidle_idle_call.constprop.0+0x237/0x410 [ 13.947610][ C0] ? lockdep_hardirqs_on+0x91/0x130 [ 13.947612][ C0] default_idle+0x9/0x10 [ 13.947613][ C0] default_idle_call+0x6e/0xb0 [ 13.947614][ C0] cpuidle_idle_call.constprop.0+0x237/0x410 [ 13.947616][ C0] ? arch_cpu_idle_exit+0x40/0x40 [ 13.947618][ C0] ? mark_tsc_async_resets+0x30/0x30 [ 13.947619][ C0] ? rcu_is_watching+0x16/0xd0 [ 13.947621][ C0] do_idle+0xd8/0x190 [ 13.947623][ C0] cpu_startup_entry+0x53/0x70 [ 13.947624][ C0] rest_init+0x279/0x280 [ 13.947626][ C0] ? __cpuidle_text_end+0x3b/0x3b [ 13.947628][ C0] ? rest_init+0x280/0x280 [ 13.947630][ C0] ? acpi_hw_set_mode+0x4a0/0x4a0 [ 13.947631][ C0] ? cpus_read_unlock+0x4f/0xc0 [ 13.947633][ C0] ? acpi_enable+0x1e4/0x330 [ 13.947635][ C0] start_kernel+0x3af/0x3b0 [ 13.947637][ C0] x86_64_start_reservations+0x24/0x30 [ 13.947639][ C0] x86_64_start_kernel+0x12b/0x130 [ 13.947641][ C0] common_startup_64+0x13e/0x148 [ 13.947644][ C0] [ 19.000729][ T70] RDS: shutdown called in state 3 [ 19.000745][ T70] BUG: sleeping function called from invalid context at ./include/linux/sched/mm.h:322 [ 19.000747][ T70] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 70, name: kworker/u16:1 [ 19.000749][ T70] preempt_count: 1, expected: 0 [ 19.000750][ T70] RCU nest depth: 0, expected: 0 [ 19.000751][ T70] INFO: lockdep is turned off. [ 19.000752][ T70] irq event stamp: 13346 [ 19.000753][ T70] hardirqs last enabled at (13345): [] _raw_spin_unlock_irq+0x28/0x50 [ 19.000762][ T70] hardirqs last disabled at (13346): [] __schedule+0x1246/0x20e0 [ 19.000765][ T70] softirqs last enabled at (13342): [] rds_send_xmit+0x1198/0x26e0 [ 19.000768][ T70] softirqs last disabled at (13340): [] release_sock+0x21/0x240 [ 19.000772][ T70] Preemption disabled at: [ 19.000772][ T70] [] vprintk_emit+0x31b/0x3e0 [ 19.000781][ T70] CPU: 2 UID: 0 PID: 70 Comm: kworker/u16:1 Tainted: G W 7.2.0-virtme #1 PREEMPT(full) [ 19.000784][ T70] Tainted: [W]=WARN [ 19.000785][ T70] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 19.000787][ T70] Workqueue: krds_cp_wq#1/5 rds_shutdown_worker [ 19.000792][ T70] Call Trace: [ 19.000793][ T70] [ 19.000795][ T70] dump_stack_lvl+0x6f/0xa0 [ 19.000800][ T70] ? vprintk_emit+0x31b/0x3e0 [ 19.000802][ T70] __might_resched.cold+0x1fe/0x2c1 [ 19.000806][ T70] ? perf_trace_sched_switch+0x7c0/0x7c0 [ 19.000810][ T70] ? prb_next_reserve_seq+0x2d0/0x2d0 [ 19.000812][ T70] ? rcu_is_watching+0x16/0xd0 [ 19.000816][ T70] __kmalloc_noprof+0x443/0x760 [ 19.000819][ T70] ? __rwlock_init+0x150/0x150 [ 19.000822][ T70] ? alloc_buf.isra.0+0x4b/0x260 [ 19.000826][ T70] ? do_raw_spin_unlock+0x59/0x250 [ 19.000828][ T70] alloc_buf.isra.0+0x4b/0x260 [ 19.000830][ T70] put_chars+0x1e1/0x2f0 [ 19.000832][ T70] ? prb_final_commit+0x50/0x50 [ 19.000834][ T70] ? __send_to_port+0x420/0x420 [ 19.000835][ T70] ? printk_get_next_message+0x240/0x650 [ 19.000838][ T70] hvc_console_print+0x292/0x780 [ 19.000842][ T70] ? hvc_write+0x3a0/0x3a0 [ 19.000844][ T70] ? do_raw_spin_lock+0x131/0x280 [ 19.000845][ T70] ? vsnprintf+0x668/0x1700 [ 19.000848][ T70] ? __rwlock_init+0x150/0x150 [ 19.000849][ T70] ? rcu_is_watching+0x16/0xd0 [ 19.000851][ T70] ? lock_acquire+0x13c/0x160 [ 19.000854][ T70] console_emit_next_record+0x22f/0x4c0 [ 19.000856][ T70] ? devkmsg_read+0x4b0/0x4b0 [ 19.000859][ T70] ? rcu_is_watching+0x16/0xd0 [ 19.000860][ T70] ? rcu_is_watching+0x16/0xd0 [ 19.000862][ T70] ? rcu_is_watching+0x16/0xd0 [ 19.000864][ T70] ? lock_acquire+0x13c/0x160 [ 19.000866][ T70] ? console_flush_one_record+0x111/0x710 [ 19.000868][ T70] console_flush_one_record+0x46f/0x710 [ 19.000870][ T70] ? console_emit_next_record+0x4c0/0x4c0 [ 19.000874][ T70] console_unlock+0xee/0x1f0 [ 19.000876][ T70] ? lock_acquire+0x13c/0x160 [ 19.000877][ T70] ? console_flush_one_record+0x710/0x710 [ 19.000879][ T70] ? rcu_is_watching+0x16/0xd0 [ 19.000881][ T70] ? lock_release+0x184/0x1f0 [ 19.000883][ T70] ? lock_acquire+0x60/0x160 [ 19.000884][ T70] ? __rcu_read_lock+0x30/0x70 [ 19.000886][ T70] ? __down_trylock_console_sem+0x5e/0xa0 [ 19.000888][ T70] ? vprintk_emit+0x320/0x3e0 [ 19.000891][ T70] vprintk_emit+0x37c/0x3e0 [ 19.000893][ T70] ? wake_up_klogd_work_func+0x90/0x90 [ 19.000895][ T70] ? lock_release+0x184/0x1f0 [ 19.000897][ T70] ? ww_mutex_lock+0x160/0x160 [ 19.000899][ T70] ? __queue_work+0x3ab/0xd30 [ 19.000902][ T70] __rds_conn_path_error+0xbd/0x100 [ 19.000905][ T70] ? rds_check_all_paths+0xb0/0xb0 [ 19.000907][ T70] ? lock_acquire+0x13c/0x160 [ 19.000909][ T70] ? rcu_is_watching+0x16/0xd0 [ 19.000911][ T70] ? rds_conn_shutdown+0x652/0xa20 [ 19.000913][ T70] rds_conn_shutdown+0x695/0xa20 [ 19.000914][ T70] ? __rds_conn_path_error+0x100/0x100 [ 19.000916][ T70] ? debug_object_active_state+0x430/0x430 [ 19.000920][ T70] ? update_load_avg+0x141/0xf00 [ 19.000922][ T70] ? rcu_is_watching+0x16/0xd0 [ 19.000924][ T70] ? rcu_is_watching+0x16/0xd0 [ 19.000926][ T70] ? lock_acquire+0x13c/0x160 [ 19.000928][ T70] ? rcu_is_watching+0x16/0xd0 [ 19.000930][ T70] ? process_one_work+0xdfc/0x1560 [ 19.000932][ T70] process_one_work+0xe3e/0x1560 [ 19.000935][ T70] ? __queue_delayed_work+0x470/0x470 [ 19.000938][ T70] ? __rwlock_init+0x150/0x150 [ 19.000940][ T70] worker_thread+0x4f1/0xd60 [ 19.000943][ T70] ? rescuer_thread+0x1340/0x1340 [ 19.000946][ T70] kthread+0x367/0x460 [ 19.000949][ T70] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 19.000952][ T70] ? kthread_affine_preferred+0x4c0/0x4c0 [ 19.000954][ T70] ret_from_fork+0x474/0x6b0 [ 19.000958][ T70] ? arch_exit_to_user_mode_prepare.isra.0+0x120/0x120 [ 19.000960][ T70] ? __switch_to+0x5a3/0xe00 [ 19.000962][ T70] ? kthread_affine_preferred+0x4c0/0x4c0 [ 19.000964][ T70] ret_from_fork_asm+0x11/0x20 [ 19.000968][ T70] [ 37.268281][ T253] RDS: shutdown called in state 1 [ 37.268298][ T253] BUG: sleeping function called from invalid context at ./include/linux/sched/mm.h:322 [ 37.268301][ T253] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 253, name: kworker/u16:14 [ 37.268302][ T253] preempt_count: 1, expected: 0 [ 37.268303][ T253] RCU nest depth: 0, expected: 0 [ 37.268304][ T253] INFO: lockdep is turned off. [ 37.268305][ T253] irq event stamp: 0 [ 37.268306][ T253] hardirqs last enabled at (0): [<0000000000000000>] 0x0 [ 37.268311][ T253] hardirqs last disabled at (0): [] copy_process+0x1ccd/0x6240 [ 37.268318][ T253] softirqs last enabled at (0): [] copy_process+0x1d21/0x6240 [ 37.268320][ T253] softirqs last disabled at (0): [<0000000000000000>] 0x0 [ 37.268321][ T253] Preemption disabled at: [ 37.268322][ T253] [] vprintk_emit+0x31b/0x3e0 [ 37.268330][ T253] CPU: 0 UID: 0 PID: 253 Comm: kworker/u16:14 Tainted: G W 7.2.0-virtme #1 PREEMPT(full) [ 37.268334][ T253] Tainted: [W]=WARN [ 37.268335][ T253] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 37.268336][ T253] Workqueue: krds_cp_wq#1/2 rds_shutdown_worker [ 37.268343][ T253] Call Trace: [ 37.268344][ T253] [ 37.268346][ T253] dump_stack_lvl+0x6f/0xa0 [ 37.268351][ T253] ? vprintk_emit+0x31b/0x3e0 [ 37.268353][ T253] __might_resched.cold+0x1fe/0x2c1 [ 37.268357][ T253] ? perf_trace_sched_switch+0x7c0/0x7c0 [ 37.268360][ T253] ? prb_next_reserve_seq+0x2d0/0x2d0 [ 37.268362][ T253] ? rcu_is_watching+0x16/0xd0 [ 37.268366][ T253] __kmalloc_noprof+0x443/0x760 [ 37.268369][ T253] ? __rwlock_init+0x150/0x150 [ 37.268372][ T253] ? alloc_buf.isra.0+0x4b/0x260 [ 37.268376][ T253] ? do_raw_spin_unlock+0x59/0x250 [ 37.268378][ T253] alloc_buf.isra.0+0x4b/0x260 [ 37.268380][ T253] put_chars+0x1e1/0x2f0 [ 37.268382][ T253] ? prb_final_commit+0x50/0x50 [ 37.268383][ T253] ? __send_to_port+0x420/0x420 [ 37.268385][ T253] ? printk_get_next_message+0x240/0x650 [ 37.268388][ T253] hvc_console_print+0x292/0x780 [ 37.268392][ T253] ? hvc_write+0x3a0/0x3a0 [ 37.268393][ T253] ? do_raw_spin_lock+0x131/0x280 [ 37.268395][ T253] ? vsnprintf+0x668/0x1700 [ 37.268397][ T253] ? __rwlock_init+0x150/0x150 [ 37.268399][ T253] ? rcu_is_watching+0x16/0xd0 [ 37.268400][ T253] ? lock_acquire+0x13c/0x160 [ 37.268403][ T253] console_emit_next_record+0x22f/0x4c0 [ 37.268405][ T253] ? devkmsg_read+0x4b0/0x4b0 [ 37.268407][ T253] ? rcu_is_watching+0x16/0xd0 [ 37.268409][ T253] ? rcu_is_watching+0x16/0xd0 [ 37.268411][ T253] ? rcu_is_watching+0x16/0xd0 [ 37.268412][ T253] ? lock_acquire+0x13c/0x160 [ 37.268414][ T253] ? console_flush_one_record+0x111/0x710 [ 37.268416][ T253] console_flush_one_record+0x46f/0x710 [ 37.268418][ T253] ? console_emit_next_record+0x4c0/0x4c0 [ 37.268421][ T253] console_unlock+0xee/0x1f0 [ 37.268423][ T253] ? lock_acquire+0x13c/0x160 [ 37.268424][ T253] ? console_flush_one_record+0x710/0x710 [ 37.268426][ T253] ? rcu_is_watching+0x16/0xd0 [ 37.268428][ T253] ? lock_release+0x184/0x1f0 [ 37.268429][ T253] ? lock_acquire+0x60/0x160 [ 37.268431][ T253] ? __rcu_read_lock+0x30/0x70 [ 37.268433][ T253] ? __down_trylock_console_sem+0x5e/0xa0 [ 37.268435][ T253] ? vprintk_emit+0x320/0x3e0 [ 37.268437][ T253] vprintk_emit+0x37c/0x3e0 [ 37.268439][ T253] ? wake_up_klogd_work_func+0x90/0x90 [ 37.268441][ T253] ? rcu_is_watching+0x16/0xd0 [ 37.268443][ T253] ? ww_mutex_lock+0x160/0x160 [ 37.268446][ T253] ? try_to_grab_pending+0x77/0x840 [ 37.268449][ T253] __rds_conn_path_error+0xbd/0x100 [ 37.268451][ T253] ? rds_check_all_paths+0xb0/0xb0 [ 37.268453][ T253] ? lock_acquire+0x13c/0x160 [ 37.268455][ T253] ? rcu_is_watching+0x16/0xd0 [ 37.268457][ T253] ? rds_conn_shutdown+0x652/0xa20 [ 37.268458][ T253] rds_conn_shutdown+0x695/0xa20 [ 37.268460][ T253] ? __rds_conn_path_error+0x100/0x100 [ 37.268462][ T253] ? debug_object_active_state+0x430/0x430 [ 37.268466][ T253] ? rcu_is_watching+0x16/0xd0 [ 37.268467][ T253] ? rcu_is_watching+0x16/0xd0 [ 37.268469][ T253] ? lock_acquire+0x13c/0x160 [ 37.268470][ T253] ? rcu_is_watching+0x16/0xd0 [ 37.268473][ T253] ? process_one_work+0xdfc/0x1560 [ 37.268474][ T253] process_one_work+0xe3e/0x1560 [ 37.268477][ T253] ? __queue_delayed_work+0x470/0x470 [ 37.268479][ T253] ? __rwlock_init+0x150/0x150 [ 37.268482][ T253] worker_thread+0x4f1/0xd60 [ 37.268484][ T253] ? rescuer_thread+0x1340/0x1340 [ 37.268486][ T253] ? __kthread_parkme+0xbd/0x210 [ 37.268488][ T253] ? rescuer_thread+0x1340/0x1340 [ 37.268490][ T253] ? rescuer_thread+0x1340/0x1340 [ 37.268491][ T253] kthread+0x367/0x460 [ 37.268493][ T253] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 37.268496][ T253] ? kthread_affine_preferred+0x4c0/0x4c0 [ 37.268498][ T253] ret_from_fork+0x474/0x6b0 [ 37.268501][ T253] ? arch_exit_to_user_mode_prepare.isra.0+0x120/0x120 [ 37.268503][ T253] ? __switch_to+0x5a3/0xe00 [ 37.268505][ T253] ? kthread_affine_preferred+0x4c0/0x4c0 [ 37.268507][ T253] ret_from_fork_asm+0x11/0x20 [ 37.268511][ T253]