[ 212.205742][ T1796] gre: GRE over IPv4 demultiplexer driver [ 212.215189][ T1796] ip_gre: GRE over IPv4 tunneling driver [ 213.910611][ C0] clocksource: Watchdog remote CPU 2 read timed out [ 286.188719][ T2045] [ 286.188803][ T2045] ======================================================== [ 286.188940][ T2045] WARNING: possible irq lock inversion dependency detected [ 286.189056][ T2045] 7.1.0-rc5-virtme #1 Not tainted [ 286.189133][ T2045] -------------------------------------------------------- [ 286.189249][ T2045] sysctl/2045 just changed the state of lock: [ 286.189347][ T2045] ffffffffb497e8b8 (acaddr_hash_lock){+.+.}-{3:3}, at: __ipv6_dev_ac_dec+0x236/0x5d0 [ 286.189493][ T2045] but this lock was taken by another, SOFTIRQ-safe lock in the past: [ 286.189611][ T2045] (&ndev->lock){++--}-{3:3} [ 286.189614][ T2045] [ 286.189614][ T2045] [ 286.189614][ T2045] and interrupts could create inverse lock ordering between them. [ 286.189614][ T2045] [ 286.189900][ T2045] [ 286.189900][ T2045] other info that might help us debug this: [ 286.190011][ T2045] Possible interrupt unsafe locking scenario: [ 286.190011][ T2045] [ 286.190125][ T2045] CPU0 CPU1 [ 286.190203][ T2045] ---- ---- [ 286.190279][ T2045] lock(acaddr_hash_lock); [ 286.190358][ T2045] local_irq_disable(); [ 286.190455][ T2045] lock(&ndev->lock); [ 286.190553][ T2045] lock(acaddr_hash_lock); [ 286.190650][ T2045] [ 286.190708][ T2045] lock(&ndev->lock); [ 286.190767][ T2045] [ 286.190767][ T2045] *** DEADLOCK *** [ 286.190767][ T2045] [ 286.190887][ T2045] 2 locks held by sysctl/2045: [ 286.190970][ T2045] #0: ff11000009c443e0 (sb_writers#3){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 286.191110][ T2045] #1: ffffffffb488c9c0 (rtnl_mutex){+.+.}-{4:4}, at: addrconf_fixup_forwarding+0x4d/0x4d0 [ 286.191269][ T2045] [ 286.191269][ T2045] the shortest dependencies between 2nd lock and 1st lock: [ 286.191406][ T2045] -> (&ndev->lock){++--}-{3:3} { [ 286.191492][ T2045] HARDIRQ-ON-W at: [ 286.191554][ T2045] __lock_acquire+0x378/0xc10 [ 286.191658][ T2045] lock_acquire.part.0+0xbc/0x260 [ 286.191757][ T2045] _raw_write_lock_bh+0x38/0x50 [ 286.191862][ T2045] addrconf_permanent_addr+0x108/0x9c0 [ 286.191980][ T2045] addrconf_notify+0x151/0xf30 [ 286.192082][ T2045] notifier_call_chain+0xb0/0x320 [ 286.192183][ T2045] __dev_notify_flags+0xde/0x280 [ 286.192283][ T2045] netif_change_flags+0xfe/0x190 [ 286.192381][ T2045] do_setlink.isra.0+0x1cc3/0x2750 [ 286.192482][ T2045] rtnl_newlink+0x8d1/0xef0 [ 286.192580][ T2045] rtnetlink_rcv_msg+0x6fd/0xbd0 [ 286.192677][ T2045] netlink_rcv_skb+0x14e/0x3a0 [ 286.192776][ T2045] netlink_unicast+0x486/0x750 [ 286.192878][ T2045] netlink_sendmsg+0x735/0xc60 [ 286.192976][ T2045] ____sys_sendmsg+0x419/0x850 [ 286.193078][ T2045] ___sys_sendmsg+0x14e/0x1d0 [ 286.193178][ T2045] __sys_sendmsg+0x145/0x1f0 [ 286.193319][ T2045] do_syscall_64+0x117/0x590 [ 286.193418][ T2045] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 286.193537][ T2045] HARDIRQ-ON-R at: [ 286.193638][ T2045] __lock_acquire+0x378/0xc10 [ 286.193735][ T2045] lock_acquire.part.0+0xbc/0x260 [ 286.193830][ T2045] _raw_read_lock_bh+0x44/0x80 [ 286.193933][ T2045] inet6_fill_ifla6_attrs+0x1aee/0x21e0 [ 286.194092][ T2045] inet6_fill_link_af+0x5c/0xe0 [ 286.194193][ T2045] rtnl_fill_ifinfo.isra.0+0x1a45/0x2c20 [ 286.194352][ T2045] rtnl_getlink+0x9c9/0xeb0 [ 286.194449][ T2045] rtnetlink_rcv_msg+0x6fd/0xbd0 [ 286.194547][ T2045] netlink_rcv_skb+0x14e/0x3a0 [ 286.194644][ T2045] netlink_unicast+0x486/0x750 [ 286.194779][ T2045] netlink_sendmsg+0x735/0xc60 [ 286.194885][ T2045] ____sys_sendmsg+0x419/0x850 [ 286.194982][ T2045] ___sys_sendmsg+0x14e/0x1d0 [ 286.195124][ T2045] __sys_sendmsg+0x145/0x1f0 [ 286.195222][ T2045] do_syscall_64+0x117/0x590 [ 286.195320][ T2045] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 286.195480][ T2045] IN-SOFTIRQ-W at: [ 286.195538][ T2045] __lock_acquire+0x378/0xc10 [ 286.195634][ T2045] lock_acquire.part.0+0xbc/0x260 [ 286.195770][ T2045] _raw_write_lock+0x33/0x40 [ 286.195872][ T2045] addrconf_rs_timer+0xb0/0x770 [ 286.195967][ T2045] call_timer_fn+0x163/0x4f0 [ 286.196066][ T2045] __run_timers+0x68f/0xab0 [ 286.196202][ T2045] run_timer_softirq+0xf0/0x160 [ 286.196300][ T2045] handle_softirqs+0x1d8/0x940 [ 286.196397][ T2045] __irq_exit_rcu+0x103/0x1c0 [ 286.196535][ T2045] irq_exit_rcu+0xe/0x30 [ 286.196633][ T2045] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 286.196751][ T2045] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 286.196911][ T2045] pv_native_safe_halt+0xf/0x10 [ 286.197007][ T2045] default_idle+0x9/0x10 [ 286.197104][ T2045] default_idle_call+0x6e/0xb0 [ 286.197241][ T2045] cpuidle_idle_call.constprop.0+0x237/0x410 [ 286.197356][ T2045] do_idle+0xf5/0x160 [ 286.197453][ T2045] cpu_startup_entry+0x53/0x70 [ 286.197548][ T2045] start_secondary+0x204/0x2b0 [ 286.197644][ T2045] common_startup_64+0x13e/0x148 [ 286.197741][ T2045] IN-SOFTIRQ-R at: [ 286.197840][ T2045] __lock_acquire+0x378/0xc10 [ 286.197940][ T2045] lock_acquire.part.0+0xbc/0x260 [ 286.198034][ T2045] _raw_read_lock_bh+0x44/0x80 [ 286.198129][ T2045] ipv6_get_lladdr+0xa8/0x3f0 [ 286.198265][ T2045] addrconf_rs_timer+0x251/0x770 [ 286.198360][ T2045] call_timer_fn+0x163/0x4f0 [ 286.198455][ T2045] __run_timers+0x68f/0xab0 [ 286.198595][ T2045] run_timer_softirq+0xf0/0x160 [ 286.198690][ T2045] handle_softirqs+0x1d8/0x940 [ 286.198784][ T2045] __irq_exit_rcu+0x103/0x1c0 [ 286.198928][ T2045] irq_exit_rcu+0xe/0x30 [ 286.199024][ T2045] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 286.199138][ T2045] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 286.199293][ T2045] pv_native_safe_halt+0xf/0x10 [ 286.199387][ T2045] default_idle+0x9/0x10 [ 286.199486][ T2045] default_idle_call+0x6e/0xb0 [ 286.199623][ T2045] cpuidle_idle_call.constprop.0+0x237/0x410 [ 286.199737][ T2045] do_idle+0xf5/0x160 [ 286.199831][ T2045] cpu_startup_entry+0x53/0x70 [ 286.199972][ T2045] start_secondary+0x204/0x2b0 [ 286.200067][ T2045] common_startup_64+0x13e/0x148 [ 286.200163][ T2045] INITIAL USE at: [ 286.200221][ T2045] __lock_acquire+0x378/0xc10 [ 286.200357][ T2045] lock_acquire.part.0+0xbc/0x260 [ 286.200452][ T2045] _raw_write_lock_bh+0x38/0x50 [ 286.200547][ T2045] addrconf_permanent_addr+0x108/0x9c0 [ 286.200703][ T2045] addrconf_notify+0x151/0xf30 [ 286.200799][ T2045] notifier_call_chain+0xb0/0x320 [ 286.200900][ T2045] __dev_notify_flags+0xde/0x280 [ 286.201036][ T2045] netif_change_flags+0xfe/0x190 [ 286.201132][ T2045] do_setlink.isra.0+0x1cc3/0x2750 [ 286.201228][ T2045] rtnl_newlink+0x8d1/0xef0 [ 286.201366][ T2045] rtnetlink_rcv_msg+0x6fd/0xbd0 [ 286.201461][ T2045] netlink_rcv_skb+0x14e/0x3a0 [ 286.201557][ T2045] netlink_unicast+0x486/0x750 [ 286.201694][ T2045] netlink_sendmsg+0x735/0xc60 [ 286.201793][ T2045] ____sys_sendmsg+0x419/0x850 [ 286.201894][ T2045] ___sys_sendmsg+0x14e/0x1d0 [ 286.202030][ T2045] __sys_sendmsg+0x145/0x1f0 [ 286.202125][ T2045] do_syscall_64+0x117/0x590 [ 286.202219][ T2045] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 286.202374][ T2045] INITIAL READ USE at: [ 286.202449][ T2045] __lock_acquire+0x378/0xc10 [ 286.202545][ T2045] lock_acquire.part.0+0xbc/0x260 [ 286.202657][ T2045] _raw_read_lock_bh+0x44/0x80 [ 286.202794][ T2045] inet6_fill_ifla6_attrs+0x1aee/0x21e0 [ 286.202915][ T2045] inet6_fill_link_af+0x5c/0xe0 [ 286.203010][ T2045] rtnl_fill_ifinfo.isra.0+0x1a45/0x2c20 [ 286.203166][ T2045] rtnl_getlink+0x9c9/0xeb0 [ 286.203261][ T2045] rtnetlink_rcv_msg+0x6fd/0xbd0 [ 286.203355][ T2045] netlink_rcv_skb+0x14e/0x3a0 [ 286.203495][ T2045] netlink_unicast+0x486/0x750 [ 286.203591][ T2045] netlink_sendmsg+0x735/0xc60 [ 286.203688][ T2045] ____sys_sendmsg+0x419/0x850 [ 286.203824][ T2045] ___sys_sendmsg+0x14e/0x1d0 [ 286.203926][ T2045] __sys_sendmsg+0x145/0x1f0 [ 286.204025][ T2045] do_syscall_64+0x117/0x590 [ 286.204163][ T2045] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 286.204277][ T2045] } [ 286.204318][ T2045] ... key at: [] __key.37+0x0/0x40 [ 286.204475][ T2045] ... acquired at: [ 286.204534][ T2045] __lock_acquire+0x508/0xc10 [ 286.204614][ T2045] lock_acquire.part.0+0xbc/0x260 [ 286.204690][ T2045] _raw_spin_lock+0x33/0x40 [ 286.204765][ T2045] ipv6_add_acaddr_hash.isra.0+0x16d/0x350 [ 286.204906][ T2045] __ipv6_dev_ac_inc+0x57e/0x930 [ 286.204981][ T2045] addrconf_join_anycast+0x1bc/0x260 [ 286.205057][ T2045] dev_forward_change+0x2be/0x850 [ 286.205175][ T2045] addrconf_fixup_forwarding+0x2af/0x4d0 [ 286.205271][ T2045] addrconf_sysctl_forward+0x209/0x2b0 [ 286.205347][ T2045] proc_sys_call_handler+0x31b/0x480 [ 286.205428][ T2045] new_sync_write+0x333/0x750 [ 286.205504][ T2045] vfs_write+0x6a4/0xc10 [ 286.205581][ T2045] ksys_write+0x116/0x250 [ 286.205657][ T2045] do_syscall_64+0x117/0x590 [ 286.205735][ T2045] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 286.205874][ T2045] [ 286.205913][ T2045] -> (acaddr_hash_lock){+.+.}-{3:3} { [ 286.205991][ T2045] HARDIRQ-ON-W at: [ 286.206050][ T2045] __lock_acquire+0x378/0xc10 [ 286.206186][ T2045] lock_acquire.part.0+0xbc/0x260 [ 286.206283][ T2045] _raw_spin_lock+0x33/0x40 [ 286.206379][ T2045] ipv6_add_acaddr_hash.isra.0+0x16d/0x350 [ 286.206535][ T2045] __ipv6_dev_ac_inc+0x57e/0x930 [ 286.206632][ T2045] addrconf_join_anycast+0x1bc/0x260 [ 286.206728][ T2045] dev_forward_change+0x2be/0x850 [ 286.206824][ T2045] addrconf_fixup_forwarding+0x2af/0x4d0 [ 286.206982][ T2045] addrconf_sysctl_forward+0x209/0x2b0 [ 286.207076][ T2045] proc_sys_call_handler+0x31b/0x480 [ 286.207172][ T2045] new_sync_write+0x333/0x750 [ 286.207307][ T2045] vfs_write+0x6a4/0xc10 [ 286.207403][ T2045] ksys_write+0x116/0x250 [ 286.207499][ T2045] do_syscall_64+0x117/0x590 [ 286.207635][ T2045] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 286.207749][ T2045] SOFTIRQ-ON-W at: [ 286.207807][ T2045] __lock_acquire+0x378/0xc10 [ 286.207950][ T2045] lock_acquire.part.0+0xbc/0x260 [ 286.208047][ T2045] _raw_spin_lock+0x33/0x40 [ 286.208144][ T2045] __ipv6_dev_ac_dec+0x236/0x5d0 [ 286.208279][ T2045] addrconf_leave_anycast+0x1bc/0x260 [ 286.208376][ T2045] dev_forward_change+0x3b3/0x850 [ 286.208473][ T2045] addrconf_fixup_forwarding+0x2af/0x4d0 [ 286.208632][ T2045] addrconf_sysctl_forward+0x209/0x2b0 [ 286.208728][ T2045] proc_sys_call_handler+0x31b/0x480 [ 286.208824][ T2045] new_sync_write+0x333/0x750 [ 286.208923][ T2045] vfs_write+0x6a4/0xc10 [ 286.209061][ T2045] ksys_write+0x116/0x250 [ 286.209157][ T2045] do_syscall_64+0x117/0x590 [ 286.209252][ T2045] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 286.209407][ T2045] INITIAL USE at: [ 286.209464][ T2045] __lock_acquire+0x378/0xc10 [ 286.209560][ T2045] lock_acquire.part.0+0xbc/0x260 [ 286.209698][ T2045] _raw_spin_lock+0x33/0x40 [ 286.209793][ T2045] ipv6_add_acaddr_hash.isra.0+0x16d/0x350 [ 286.209910][ T2045] __ipv6_dev_ac_inc+0x57e/0x930 [ 286.210046][ T2045] addrconf_join_anycast+0x1bc/0x260 [ 286.210142][ T2045] dev_forward_change+0x2be/0x850 [ 286.210240][ T2045] addrconf_fixup_forwarding+0x2af/0x4d0 [ 286.210398][ T2045] addrconf_sysctl_forward+0x209/0x2b0 [ 286.210496][ T2045] proc_sys_call_handler+0x31b/0x480 [ 286.210592][ T2045] new_sync_write+0x333/0x750 [ 286.210728][ T2045] vfs_write+0x6a4/0xc10 [ 286.210823][ T2045] ksys_write+0x116/0x250 [ 286.210923][ T2045] do_syscall_64+0x117/0x590 [ 286.211018][ T2045] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 286.211174][ T2045] } [ 286.211213][ T2045] ... key at: [] acaddr_hash_lock+0x18/0x45a0 [ 286.211328][ T2045] ... acquired at: [ 286.211425][ T2045] mark_lock+0x1d7/0xa00 [ 286.211501][ T2045] mark_usage+0x105/0x170 [ 286.211580][ T2045] __lock_acquire+0x378/0xc10 [ 286.211655][ T2045] lock_acquire.part.0+0xbc/0x260 [ 286.211770][ T2045] _raw_spin_lock+0x33/0x40 [ 286.211847][ T2045] __ipv6_dev_ac_dec+0x236/0x5d0 [ 286.211926][ T2045] addrconf_leave_anycast+0x1bc/0x260 [ 286.212003][ T2045] dev_forward_change+0x3b3/0x850 [ 286.212118][ T2045] addrconf_fixup_forwarding+0x2af/0x4d0 [ 286.212214][ T2045] addrconf_sysctl_forward+0x209/0x2b0 [ 286.212290][ T2045] proc_sys_call_handler+0x31b/0x480 [ 286.212366][ T2045] new_sync_write+0x333/0x750 [ 286.212482][ T2045] vfs_write+0x6a4/0xc10 [ 286.212560][ T2045] ksys_write+0x116/0x250 [ 286.212635][ T2045] do_syscall_64+0x117/0x590 [ 286.212712][ T2045] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 286.212847][ T2045] [ 286.212889][ T2045] [ 286.212889][ T2045] stack backtrace: [ 286.212983][ T2045] CPU: 2 UID: 0 PID: 2045 Comm: sysctl Not tainted 7.1.0-rc5-virtme #1 PREEMPT(full) [ 286.212985][ T2045] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 286.212987][ T2045] Call Trace: [ 286.212989][ T2045] [ 286.212990][ T2045] dump_stack_lvl+0x6f/0xa0 [ 286.212995][ T2045] print_irq_inversion_bug.part.0.cold+0xe6/0x143 [ 286.212998][ T2045] mark_lock_irq+0x989/0x9c0 [ 286.213001][ T2045] mark_lock+0x1d7/0xa00 [ 286.213003][ T2045] mark_usage+0x105/0x170 [ 286.213004][ T2045] __lock_acquire+0x378/0xc10 [ 286.213006][ T2045] lock_acquire.part.0+0xbc/0x260 [ 286.213008][ T2045] ? __ipv6_dev_ac_dec+0x236/0x5d0 [ 286.213009][ T2045] ? rcu_is_watching+0x15/0xd0 [ 286.213012][ T2045] ? do_raw_read_unlock+0x70/0x70 [ 286.213014][ T2045] ? lock_acquire+0x134/0x160 [ 286.213015][ T2045] _raw_spin_lock+0x33/0x40 [ 286.213017][ T2045] ? __ipv6_dev_ac_dec+0x236/0x5d0 [ 286.213018][ T2045] __ipv6_dev_ac_dec+0x236/0x5d0 [ 286.213020][ T2045] addrconf_leave_anycast+0x1bc/0x260 [ 286.213022][ T2045] ? find_held_lock+0x2b/0x80 [ 286.213025][ T2045] ? __ipv6_isatap_ifid+0x210/0x210 [ 286.213026][ T2045] ? mark_held_locks+0x40/0x70 [ 286.213028][ T2045] ? lockdep_hardirqs_on+0x8c/0x130 [ 286.213030][ T2045] dev_forward_change+0x3b3/0x850 [ 286.213032][ T2045] ? addrconf_fixup_forwarding+0x4d/0x4d0 [ 286.213034][ T2045] ? addrconf_sysctl_proxy_ndp+0x2f0/0x2f0 [ 286.213036][ T2045] addrconf_fixup_forwarding+0x2af/0x4d0 [ 286.213039][ T2045] addrconf_sysctl_forward+0x209/0x2b0 [ 286.213041][ T2045] ? addrconf_fixup_forwarding+0x4d0/0x4d0 [ 286.213043][ T2045] ? addrconf_fixup_forwarding+0x4d0/0x4d0 [ 286.213045][ T2045] ? __kvmalloc_node_noprof+0x305/0x8a0 [ 286.213048][ T2045] ? proc_sys_call_handler+0x25d/0x480 [ 286.213050][ T2045] proc_sys_call_handler+0x31b/0x480 [ 286.213052][ T2045] ? proc_sys_lookup+0x3d0/0x3d0 [ 286.213054][ T2045] ? rcu_read_unlock+0x1b/0x70 [ 286.213057][ T2045] ? do_raw_spin_unlock+0x59/0x250 [ 286.213058][ T2045] ? rcu_lockdep_current_cpu_online+0x39/0x1b0 [ 286.213060][ T2045] ? rcu_read_lock_any_held+0x3c/0x90 [ 286.213062][ T2045] ? proc_sys_call_handler+0x480/0x480 [ 286.213064][ T2045] new_sync_write+0x333/0x750 [ 286.213066][ T2045] ? __lock_acquire+0x508/0xc10 [ 286.213067][ T2045] ? new_sync_read+0x740/0x740 [ 286.213069][ T2045] ? lock_acquire.part.0+0xbc/0x260 [ 286.213070][ T2045] ? ksys_write+0x116/0x250 [ 286.213072][ T2045] vfs_write+0x6a4/0xc10 [ 286.213074][ T2045] ksys_write+0x116/0x250 [ 286.213075][ T2045] ? __ia32_sys_read+0xc0/0xc0 [ 286.213077][ T2045] ? rcu_is_watching+0x15/0xd0 [ 286.213078][ T2045] ? rcu_is_watching+0x15/0xd0 [ 286.213080][ T2045] do_syscall_64+0x117/0x590 [ 286.213081][ T2045] ? trace_hardirqs_off+0xd/0x30 [ 286.213084][ T2045] ? exc_page_fault+0xee/0x100 [ 286.213086][ T2045] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 286.213088][ T2045] RIP: 0033:0x7fb2c657008e [ 286.213090][ T2045] Code: 4d 89 d8 e8 94 bd 00 00 4c 8b 5d f8 41 8b 93 08 03 00 00 59 5e 48 83 f8 fc 74 11 c9 c3 0f 1f 80 00 00 00 00 48 8b 45 10 0f 05 c3 83 e2 39 83 fa 08 75 e7 e8 03 ff ff ff 0f 1f 00 f3 0f 1e fa [ 286.213092][ T2045] RSP: 002b:00007ffd1cbf4e70 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 286.213095][ T2045] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fb2c657008e [ 286.213096][ T2045] RDX: 0000000000000002 RSI: 00005625d11da4b0 RDI: 0000000000000005 [ 286.213097][ T2045] RBP: 00007ffd1cbf4e80 R08: 0000000000000000 R09: 0000000000000000 [ 286.213098][ T2045] R10: 0000000000000000 R11: 0000000000000202 R12: 00005625d11dc5e0 [ 286.213098][ T2045] R13: 00005625d11da470 R14: 0000000000000002 R15: 0000000000000000 [ 286.213101][ T2045]