[ 40.993426][ C0] clocksource: Marking clocksource tsc unstable due to frequency skew [ 40.993462][ C0] [ 40.993463][ C0] ======================================================== [ 40.993465][ C0] WARNING: possible irq lock inversion dependency detected [ 40.993467][ C0] 7.2.0-virtme #1 Tainted: G W [ 40.993469][ C0] -------------------------------------------------------- [ 40.993469][ C0] ksoftirqd/0/14 just changed the state of lock: [ 40.993471][ C0] ffffffff99a89660 (console_owner){..-.}-{0:0}, at: console_lock_spinning_enable+0x40/0x60 [ 40.993483][ C0] but this lock took another, SOFTIRQ-unsafe lock in the past: [ 40.993484][ C0] (fs_reclaim){+.+.}-{0:0} [ 40.993486][ C0] [ 40.993486][ C0] [ 40.993486][ C0] and interrupts could create inverse lock ordering between them. [ 40.993486][ C0] [ 40.993487][ C0] [ 40.993487][ C0] other info that might help us debug this: [ 40.993487][ C0] Possible interrupt unsafe locking scenario: [ 40.993487][ C0] [ 40.993488][ C0] CPU0 CPU1 [ 40.993488][ C0] ---- ---- [ 40.993488][ C0] lock(fs_reclaim); [ 40.993489][ C0] local_irq_disable(); [ 40.993490][ C0] lock(console_owner); [ 40.993491][ C0] lock(fs_reclaim); [ 40.993491][ C0] [ 40.993492][ C0] lock(console_owner); [ 40.993493][ C0] [ 40.993493][ C0] *** DEADLOCK *** [ 40.993493][ C0] [ 40.993493][ C0] locks held by ksoftirqd/0/14: 4, last CPU#0: [ 40.993494][ C0] #0: ffa00000000e7ab8 ((&watchdog_timer)){+.-.}-{0:0}, at: call_timer_fn+0x110/0x4d0 [ 40.993500][ C0] #1: ffffffff99bce8b8 (watchdog_lock){+.-.}-{3:3}, at: clocksource_watchdog+0x15/0x30 [ 40.993504][ C0] #2: ffffffff99b69cc0 (console_lock){+.+.}-{0:0}, at: vprintk_emit+0x320/0x3e0 [ 40.993507][ C0] #3: ffffffff99b69d38 (console_srcu){....}-{0:0}, at: console_flush_one_record+0x106/0x710 [ 40.993511][ C0] [ 40.993511][ C0] the shortest dependencies between 2nd lock and 1st lock: [ 40.993514][ C0] -> (fs_reclaim){+.+.}-{0:0} { [ 40.993516][ C0] HARDIRQ-ON-W at: [ 40.993518][ C0] __lock_acquire+0x388/0xc20 [ 40.993521][ C0] lock_acquire.part.0+0xd4/0x280 [ 40.993522][ C0] fs_reclaim_acquire+0xd5/0x120 [ 40.993525][ C0] __kmalloc_cache_noprof+0x6e/0x620 [ 40.993527][ C0] kthread_create_worker_on_node+0xea/0x210 [ 40.993530][ C0] workqueue_init+0x2a/0x680 [ 40.993534][ C0] kernel_init_freeable+0x2fe/0x630 [ 40.993536][ C0] kernel_init+0x21/0x150 [ 40.993540][ C0] ret_from_fork+0x474/0x6b0 [ 40.993543][ C0] ret_from_fork_asm+0x11/0x20 [ 40.993546][ C0] SOFTIRQ-ON-W at: [ 40.993547][ C0] __lock_acquire+0x388/0xc20 [ 40.993548][ C0] lock_acquire.part.0+0xd4/0x280 [ 40.993550][ C0] fs_reclaim_acquire+0xd5/0x120 [ 40.993551][ C0] __kmalloc_cache_noprof+0x6e/0x620 [ 40.993552][ C0] kthread_create_worker_on_node+0xea/0x210 [ 40.993554][ C0] workqueue_init+0x2a/0x680 [ 40.993555][ C0] kernel_init_freeable+0x2fe/0x630 [ 40.993556][ C0] kernel_init+0x21/0x150 [ 40.993557][ C0] ret_from_fork+0x474/0x6b0 [ 40.993559][ C0] ret_from_fork_asm+0x11/0x20 [ 40.993560][ C0] INITIAL USE at: [ 40.993561][ C0] __lock_acquire+0x388/0xc20 [ 40.993562][ C0] lock_acquire.part.0+0xd4/0x280 [ 40.993563][ C0] fs_reclaim_acquire+0xd5/0x120 [ 40.993565][ C0] __kmalloc_cache_noprof+0x6e/0x620 [ 40.993566][ C0] kthread_create_worker_on_node+0xea/0x210 [ 40.993567][ C0] workqueue_init+0x2a/0x680 [ 40.993568][ C0] kernel_init_freeable+0x2fe/0x630 [ 40.993569][ C0] kernel_init+0x21/0x150 [ 40.993571][ C0] ret_from_fork+0x474/0x6b0 [ 40.993572][ C0] ret_from_fork_asm+0x11/0x20 [ 40.993573][ C0] } [ 40.993574][ C0] ... key at: [] __fs_reclaim_map+0x0/0xbe0 [ 40.993578][ C0] ... acquired at: [ 40.993579][ C0] __lock_acquire+0x518/0xc20 [ 40.993580][ C0] lock_acquire.part.0+0xd4/0x280 [ 40.993581][ C0] fs_reclaim_acquire+0xd5/0x120 [ 40.993582][ C0] __kmalloc_noprof+0xd3/0x760 [ 40.993584][ C0] alloc_buf.isra.0+0x4b/0x260 [ 40.993587][ C0] put_chars+0x1e1/0x2f0 [ 40.993588][ C0] hvc_console_print+0x292/0x780 [ 40.993590][ C0] console_emit_next_record+0x22f/0x4c0 [ 40.993593][ C0] console_flush_one_record+0x46f/0x710 [ 40.993595][ C0] console_unlock+0xee/0x1f0 [ 40.993596][ C0] vprintk_emit+0x37c/0x3e0 [ 40.993598][ C0] dev_vprintk_emit+0x27f/0x2c0 [ 40.993601][ C0] dev_printk_emit+0xb9/0xee [ 40.993603][ C0] _dev_info+0xe2/0x116 [ 40.993605][ C0] __devm_rtc_register_device.cold+0xc3/0x3a8 [ 40.993607][ C0] cmos_do_probe+0x73b/0x98a [ 40.993609][ C0] platform_probe+0xfe/0x1f0 [ 40.993612][ C0] call_driver_probe+0x61/0x1c0 [ 40.993614][ C0] really_probe+0x199/0x760 [ 40.993616][ C0] __driver_probe_device+0x24f/0x440 [ 40.993617][ C0] driver_probe_device+0x4a/0xf0 [ 40.993618][ C0] __driver_attach+0x1b8/0x540 [ 40.993620][ C0] bus_for_each_dev+0x130/0x1e0 [ 40.993621][ C0] bus_add_driver+0x2c8/0x530 [ 40.993622][ C0] driver_register+0x1a3/0x390 [ 40.993624][ C0] __platform_driver_probe+0x13f/0x270 [ 40.993625][ C0] cmos_init+0x31/0x40 [ 40.993629][ C0] do_one_initcall+0x124/0x4f0 [ 40.993630][ C0] kernel_init_freeable+0x596/0x630 [ 40.993631][ C0] kernel_init+0x21/0x150 [ 40.993632][ C0] ret_from_fork+0x474/0x6b0 [ 40.993634][ C0] ret_from_fork_asm+0x11/0x20 [ 40.993635][ C0] [ 40.993635][ C0] -> (console_owner){..-.}-{0:0} { [ 40.993637][ C0] IN-SOFTIRQ-W at: [ 40.993638][ C0] __lock_acquire+0x388/0xc20 [ 40.993639][ C0] lock_acquire.part.0+0xd4/0x280 [ 40.993640][ C0] console_lock_spinning_enable+0x5c/0x60 [ 40.993642][ C0] console_emit_next_record+0x1d1/0x4c0 [ 40.993644][ C0] console_flush_one_record+0x46f/0x710 [ 40.993645][ C0] console_unlock+0xee/0x1f0 [ 40.993647][ C0] vprintk_emit+0x37c/0x3e0 [ 40.993648][ C0] _printk+0xc7/0x100 [ 40.993651][ C0] clocksource_watchdog.part.0.cold+0x32/0x1dd [ 40.993654][ C0] call_timer_fn+0x160/0x4d0 [ 40.993655][ C0] __run_timers+0x68f/0xaa0 [ 40.993657][ C0] run_timer_softirq+0xf0/0x160 [ 40.993658][ C0] handle_softirqs+0x1d3/0x900 [ 40.993661][ C0] run_ksoftirqd+0x39/0x60 [ 40.993662][ C0] smpboot_thread_fn+0x2fb/0x9b0 [ 40.993664][ C0] kthread+0x367/0x460 [ 40.993665][ C0] ret_from_fork+0x474/0x6b0 [ 40.993666][ C0] ret_from_fork_asm+0x11/0x20 [ 40.993668][ C0] INITIAL USE at: [ 40.993669][ C0] } [ 40.993669][ C0] ... key at: [] console_owner_dep_map+0x0/0x60 [ 40.993672][ C0] ... acquired at: [ 40.993672][ C0] mark_lock+0x1d7/0xa00 [ 40.993674][ C0] mark_usage+0x42/0x170 [ 40.993675][ C0] __lock_acquire+0x388/0xc20 [ 40.993676][ C0] lock_acquire.part.0+0xd4/0x280 [ 40.993678][ C0] console_lock_spinning_enable+0x5c/0x60 [ 40.993679][ C0] console_emit_next_record+0x1d1/0x4c0 [ 40.993681][ C0] console_flush_one_record+0x46f/0x710 [ 40.993682][ C0] console_unlock+0xee/0x1f0 [ 40.993684][ C0] vprintk_emit+0x37c/0x3e0 [ 40.993685][ C0] _printk+0xc7/0x100 [ 40.993687][ C0] clocksource_watchdog.part.0.cold+0x32/0x1dd [ 40.993688][ C0] call_timer_fn+0x160/0x4d0 [ 40.993689][ C0] __run_timers+0x68f/0xaa0 [ 40.993691][ C0] run_timer_softirq+0xf0/0x160 [ 40.993692][ C0] handle_softirqs+0x1d3/0x900 [ 40.993693][ C0] run_ksoftirqd+0x39/0x60 [ 40.993694][ C0] smpboot_thread_fn+0x2fb/0x9b0 [ 40.993696][ C0] kthread+0x367/0x460 [ 40.993697][ C0] ret_from_fork+0x474/0x6b0 [ 40.993698][ C0] ret_from_fork_asm+0x11/0x20 [ 40.993699][ C0] [ 40.993700][ C0] [ 40.993700][ C0] stack backtrace: [ 40.993703][ C0] CPU: 0 UID: 0 PID: 14 Comm: ksoftirqd/0 Tainted: G W 7.2.0-virtme #1 PREEMPT(full) [ 40.993706][ C0] Tainted: [W]=WARN [ 40.993707][ C0] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 40.993709][ C0] Call Trace: [ 40.993710][ C0] [ 40.993711][ C0] dump_stack_lvl+0x6f/0xa0 [ 40.993715][ C0] print_irq_inversion_bug.part.0.cold+0xe6/0x143 [ 40.993718][ C0] mark_lock_irq+0x989/0x9c0 [ 40.993720][ C0] mark_lock+0x1d7/0xa00 [ 40.993722][ C0] mark_usage+0x42/0x170 [ 40.993724][ C0] __lock_acquire+0x388/0xc20 [ 40.993726][ C0] lock_acquire.part.0+0xd4/0x280 [ 40.993728][ C0] ? console_lock_spinning_enable+0x40/0x60 [ 40.993730][ C0] ? rcu_is_watching+0x16/0xd0 [ 40.993733][ C0] ? lock_acquire+0x13c/0x160 [ 40.993735][ C0] console_lock_spinning_enable+0x5c/0x60 [ 40.993736][ C0] ? console_lock_spinning_enable+0x40/0x60 [ 40.993738][ C0] console_emit_next_record+0x1d1/0x4c0 [ 40.993740][ C0] ? devkmsg_read+0x4b0/0x4b0 [ 40.993742][ C0] ? console_flush_one_record+0x106/0x710 [ 40.993744][ C0] ? rcu_is_watching+0x16/0xd0 [ 40.993746][ C0] ? lock_acquire+0x13c/0x160 [ 40.993747][ C0] console_flush_one_record+0x46f/0x710 [ 40.993750][ C0] ? console_emit_next_record+0x4c0/0x4c0 [ 40.993751][ C0] ? __lock_acquire+0x518/0xc20 [ 40.993753][ C0] console_unlock+0xee/0x1f0 [ 40.993755][ C0] ? console_flush_one_record+0x710/0x710 [ 40.993757][ C0] ? rcu_is_watching+0x16/0xd0 [ 40.993759][ C0] ? lock_acquire+0x60/0x160 [ 40.993761][ C0] ? __down_trylock_console_sem+0x5e/0xa0 [ 40.993762][ C0] ? vprintk_emit+0x320/0x3e0 [ 40.993764][ C0] vprintk_emit+0x37c/0x3e0 [ 40.993766][ C0] ? wake_up_klogd_work_func+0x90/0x90 [ 40.993768][ C0] ? rcu_read_lock_any_held+0x3c/0x90 [ 40.993770][ C0] ? validate_chain+0x38b/0xc20 [ 40.993772][ C0] ? validate_chain+0x38b/0xc20 [ 40.993773][ C0] ? clocksource_watchdog.part.0+0x450/0x540 [ 40.993775][ C0] _printk+0xc7/0x100 [ 40.993777][ C0] ? snapshot_read.cold+0x21/0x21 [ 40.993779][ C0] ? lockdep_hardirqs_on+0x91/0x130 [ 40.993781][ C0] ? watchdog_check_freq+0x2d5/0x5c0 [ 40.993784][ C0] clocksource_watchdog.part.0.cold+0x32/0x1dd [ 40.993785][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 40.993787][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 40.993788][ C0] call_timer_fn+0x160/0x4d0 [ 40.993790][ C0] ? detach_if_pending+0x1d0/0x1d0 [ 40.993792][ C0] ? find_held_lock+0x2b/0x80 [ 40.993794][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 40.993796][ C0] ? rcu_is_watching+0x16/0xd0 [ 40.993798][ C0] __run_timers+0x68f/0xaa0 [ 40.993800][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 40.993802][ C0] ? __bpf_trace_itimer_expire+0x10/0x10 [ 40.993804][ C0] ? __lock_acquire+0x518/0xc20 [ 40.993806][ C0] ? __rwlock_init+0x150/0x150 [ 40.993809][ C0] run_timer_softirq+0xf0/0x160 [ 40.993811][ C0] ? __run_timers+0xaa0/0xaa0 [ 40.993812][ C0] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 40.993815][ C0] ? trace_irq_disable.constprop.0+0x9b/0x160 [ 40.993816][ C0] ? rcu_is_watching+0x16/0xd0 [ 40.993818][ C0] handle_softirqs+0x1d3/0x900 [ 40.993820][ C0] ? _local_bh_enable+0xc0/0xc0 [ 40.993822][ C0] ? rcu_is_watching+0x16/0xd0 [ 40.993824][ C0] run_ksoftirqd+0x39/0x60 [ 40.993825][ C0] smpboot_thread_fn+0x2fb/0x9b0 [ 40.993827][ C0] ? sort_range+0x20/0x20 [ 40.993829][ C0] kthread+0x367/0x460 [ 40.993830][ C0] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 40.993831][ C0] ? kthread_affine_preferred+0x4c0/0x4c0 [ 40.993833][ C0] ret_from_fork+0x474/0x6b0 [ 40.993835][ C0] ? arch_exit_to_user_mode_prepare.isra.0+0x120/0x120 [ 40.993837][ C0] ? __switch_to+0x5a3/0xe00 [ 40.993839][ C0] ? kthread_affine_preferred+0x4c0/0x4c0 [ 40.993841][ C0] ret_from_fork_asm+0x11/0x20 [ 40.993844][ C0] [ 40.993848][ C0] BUG: sleeping function called from invalid context at ./include/linux/sched/mm.h:322 [ 40.993849][ C0] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 14, name: ksoftirqd/0 [ 40.993851][ C0] preempt_count: 102, expected: 0 [ 40.993851][ C0] RCU nest depth: 0, expected: 0 [ 40.993852][ C0] INFO: lockdep is turned off. [ 40.993853][ C0] irq event stamp: 2124675 [ 40.993853][ C0] hardirqs last enabled at (2124674): [] __down_trylock_console_sem+0x86/0xa0 [ 40.993861][ C0] hardirqs last disabled at (2124675): [] console_emit_next_record+0x3d4/0x4c0 [ 40.993863][ C0] softirqs last enabled at (1166410): [] handle_softirqs+0x67c/0x900 [ 40.993864][ C0] softirqs last disabled at (1166417): [] run_ksoftirqd+0x39/0x60 [ 40.993865][ C0] Preemption disabled at: [ 40.993866][ C0] [] handle_softirqs+0xf6/0x900 [ 40.993869][ C0] CPU: 0 UID: 0 PID: 14 Comm: ksoftirqd/0 Tainted: G W 7.2.0-virtme #1 PREEMPT(full) [ 40.993871][ C0] Tainted: [W]=WARN [ 40.993871][ C0] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 40.993872][ C0] Call Trace: [ 40.993872][ C0] [ 40.993873][ C0] dump_stack_lvl+0x6f/0xa0 [ 40.993875][ C0] ? handle_softirqs+0xf6/0x900 [ 40.993876][ C0] __might_resched.cold+0x1fe/0x2c1 [ 40.993879][ C0] ? perf_trace_sched_switch+0x7c0/0x7c0 [ 40.993882][ C0] ? nbcon_get_cpu_emergency_nesting+0xa/0x30 [ 40.993884][ C0] ? rcu_is_watching+0x16/0xd0 [ 40.993886][ C0] __kmalloc_noprof+0x443/0x760 [ 40.993888][ C0] ? __rwlock_init+0x150/0x150 [ 40.993889][ C0] ? alloc_buf.isra.0+0x4b/0x260 [ 40.993891][ C0] ? do_raw_spin_unlock+0x59/0x250 [ 40.993892][ C0] alloc_buf.isra.0+0x4b/0x260 [ 40.993894][ C0] put_chars+0x1e1/0x2f0 [ 40.993896][ C0] ? __send_to_port+0x420/0x420 [ 40.993897][ C0] ? mark_lock+0x1d7/0xa00 [ 40.993899][ C0] hvc_console_print+0x292/0x780 [ 40.993902][ C0] ? hvc_write+0x3a0/0x3a0 [ 40.993903][ C0] ? lock_acquire+0x13c/0x160 [ 40.993905][ C0] console_emit_next_record+0x22f/0x4c0 [ 40.993907][ C0] ? devkmsg_read+0x4b0/0x4b0 [ 40.993909][ C0] ? console_flush_one_record+0x106/0x710 [ 40.993911][ C0] ? rcu_is_watching+0x16/0xd0 [ 40.993912][ C0] ? lock_acquire+0x13c/0x160 [ 40.993914][ C0] console_flush_one_record+0x46f/0x710 [ 40.993916][ C0] ? console_emit_next_record+0x4c0/0x4c0 [ 40.993918][ C0] ? __lock_acquire+0x518/0xc20 [ 40.993920][ C0] console_unlock+0xee/0x1f0 [ 40.993922][ C0] ? console_flush_one_record+0x710/0x710 [ 40.993924][ C0] ? rcu_is_watching+0x16/0xd0 [ 40.993925][ C0] ? lock_acquire+0x60/0x160 [ 40.993927][ C0] ? __down_trylock_console_sem+0x5e/0xa0 [ 40.993929][ C0] ? vprintk_emit+0x320/0x3e0 [ 40.993930][ C0] vprintk_emit+0x37c/0x3e0 [ 40.993933][ C0] ? wake_up_klogd_work_func+0x90/0x90 [ 40.993934][ C0] ? rcu_read_lock_any_held+0x3c/0x90 [ 40.993936][ C0] ? validate_chain+0x38b/0xc20 [ 40.993937][ C0] ? validate_chain+0x38b/0xc20 [ 40.993939][ C0] ? clocksource_watchdog.part.0+0x450/0x540 [ 40.993940][ C0] _printk+0xc7/0x100 [ 40.993942][ C0] ? snapshot_read.cold+0x21/0x21 [ 40.993944][ C0] ? lockdep_hardirqs_on+0x91/0x130 [ 40.993945][ C0] ? watchdog_check_freq+0x2d5/0x5c0 [ 40.993948][ C0] clocksource_watchdog.part.0.cold+0x32/0x1dd [ 40.993949][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 40.993951][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 40.993952][ C0] call_timer_fn+0x160/0x4d0 [ 40.993954][ C0] ? detach_if_pending+0x1d0/0x1d0 [ 40.993956][ C0] ? find_held_lock+0x2b/0x80 [ 40.993957][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 40.993959][ C0] ? rcu_is_watching+0x16/0xd0 [ 40.993961][ C0] __run_timers+0x68f/0xaa0 [ 40.993963][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 40.993965][ C0] ? __bpf_trace_itimer_expire+0x10/0x10 [ 40.993967][ C0] ? __lock_acquire+0x518/0xc20 [ 40.993969][ C0] ? __rwlock_init+0x150/0x150 [ 40.993971][ C0] run_timer_softirq+0xf0/0x160 [ 40.993973][ C0] ? __run_timers+0xaa0/0xaa0 [ 40.993975][ C0] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 40.993976][ C0] ? trace_irq_disable.constprop.0+0x9b/0x160 [ 40.993977][ C0] ? rcu_is_watching+0x16/0xd0 [ 40.993979][ C0] handle_softirqs+0x1d3/0x900 [ 40.993981][ C0] ? _local_bh_enable+0xc0/0xc0 [ 40.993982][ C0] ? rcu_is_watching+0x16/0xd0 [ 40.993984][ C0] run_ksoftirqd+0x39/0x60 [ 40.993986][ C0] smpboot_thread_fn+0x2fb/0x9b0 [ 40.993988][ C0] ? sort_range+0x20/0x20 [ 40.993989][ C0] kthread+0x367/0x460 [ 40.993991][ C0] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 40.993992][ C0] ? kthread_affine_preferred+0x4c0/0x4c0 [ 40.993993][ C0] ret_from_fork+0x474/0x6b0 [ 40.993995][ C0] ? arch_exit_to_user_mode_prepare.isra.0+0x120/0x120 [ 40.993996][ C0] ? __switch_to+0x5a3/0xe00 [ 40.993998][ C0] ? kthread_affine_preferred+0x4c0/0x4c0 [ 40.994000][ C0] ret_from_fork_asm+0x11/0x20 [ 40.994002][ C0] [ 41.036383][ C0] clocksource: Watchdog kvm-clock interval: 11250ns [ 41.036582][ C0] clocksource: Clocksource tsc interval: 11100ns [ 41.036781][ C0] tsc: Marking TSC unstable due to clocksource watchdog