[ 9.403018][ T214] 8021q: 802.1Q VLAN Support v1.8 [ 10.287799][ T250] br1: port 1(veth1) entered blocking state [ 10.288052][ T250] br1: port 1(veth1) entered disabled state [ 10.288227][ T250] veth1: entered allmulticast mode [ 10.289589][ T250] veth1: entered promiscuous mode [ 10.318428][ T46] br1: port 1(veth1) entered blocking state [ 10.318651][ T46] br1: port 1(veth1) entered forwarding state [ 32.403808][ C0] clocksource: Watchdog remote CPU 3 read timed out [ 36.619594][ T423] [ 36.619681][ T423] ======================================================== [ 36.619810][ T423] WARNING: possible irq lock inversion dependency detected [ 36.619931][ T423] 7.1.0-rc5-virtme #1 Not tainted [ 36.620014][ T423] -------------------------------------------------------- [ 36.620137][ T423] sysctl/423 just changed the state of lock: [ 36.620233][ T423] ffffffffb257e8b8 (acaddr_hash_lock){+.+.}-{3:3}, at: __ipv6_dev_ac_dec+0x236/0x5d0 [ 36.620379][ T423] but this lock was taken by another, SOFTIRQ-READ-safe lock in the past: [ 36.620503][ T423] (&ndev->lock){++.-}-{3:3} [ 36.620506][ T423] [ 36.620506][ T423] [ 36.620506][ T423] and interrupts could create inverse lock ordering between them. [ 36.620506][ T423] [ 36.620807][ T423] [ 36.620807][ T423] other info that might help us debug this: [ 36.620923][ T423] Possible interrupt unsafe locking scenario: [ 36.620923][ T423] [ 36.621048][ T423] CPU0 CPU1 [ 36.621136][ T423] ---- ---- [ 36.621221][ T423] lock(acaddr_hash_lock); [ 36.621306][ T423] local_irq_disable(); [ 36.621407][ T423] lock(&ndev->lock); [ 36.621507][ T423] lock(acaddr_hash_lock); [ 36.621605][ T423] [ 36.621663][ T423] lock(&ndev->lock); [ 36.621724][ T423] [ 36.621724][ T423] *** DEADLOCK *** [ 36.621724][ T423] [ 36.621841][ T423] 2 locks held by sysctl/423: [ 36.621921][ T423] #0: ff110000099b43e0 (sb_writers#3){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 36.622069][ T423] #1: ffffffffb248c9c0 (rtnl_mutex){+.+.}-{4:4}, at: addrconf_fixup_forwarding+0x4d/0x4d0 [ 36.622230][ T423] [ 36.622230][ T423] the shortest dependencies between 2nd lock and 1st lock: [ 36.622362][ T423] -> (&ndev->lock){++.-}-{3:3} { [ 36.622445][ T423] HARDIRQ-ON-W at: [ 36.622510][ T423] __lock_acquire+0x378/0xc10 [ 36.622611][ T423] lock_acquire.part.0+0xbc/0x260 [ 36.622708][ T423] _raw_write_lock_bh+0x38/0x50 [ 36.622814][ T423] addrconf_permanent_addr+0x108/0x9c0 [ 36.622929][ T423] addrconf_notify+0x151/0xf30 [ 36.623031][ T423] notifier_call_chain+0xb0/0x320 [ 36.623128][ T423] __dev_notify_flags+0xde/0x280 [ 36.623229][ T423] netif_change_flags+0xfe/0x190 [ 36.623333][ T423] do_setlink.isra.0+0x1cc3/0x2750 [ 36.623429][ T423] rtnl_newlink+0x8d1/0xef0 [ 36.623528][ T423] rtnetlink_rcv_msg+0x6fd/0xbd0 [ 36.623626][ T423] netlink_rcv_skb+0x14e/0x3a0 [ 36.623723][ T423] netlink_unicast+0x486/0x750 [ 36.623822][ T423] netlink_sendmsg+0x735/0xc60 [ 36.623919][ T423] ____sys_sendmsg+0x419/0x850 [ 36.624019][ T423] ___sys_sendmsg+0x14e/0x1d0 [ 36.624114][ T423] __sys_sendmsg+0x145/0x1f0 [ 36.624211][ T423] do_syscall_64+0x117/0x590 [ 36.624308][ T423] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 36.624423][ T423] HARDIRQ-ON-R at: [ 36.624481][ T423] __lock_acquire+0x378/0xc10 [ 36.624575][ T423] lock_acquire.part.0+0xbc/0x260 [ 36.624670][ T423] _raw_read_lock_bh+0x44/0x80 [ 36.624766][ T423] inet6_fill_ifla6_attrs+0x1aee/0x21e0 [ 36.624889][ T423] inet6_fill_link_af+0x5c/0xe0 [ 36.624985][ T423] rtnl_fill_ifinfo.isra.0+0x1a45/0x2c20 [ 36.625112][ T423] rtnl_getlink+0x9c9/0xeb0 [ 36.625209][ T423] rtnetlink_rcv_msg+0x6fd/0xbd0 [ 36.625304][ T423] netlink_rcv_skb+0x14e/0x3a0 [ 36.625400][ T423] netlink_unicast+0x486/0x750 [ 36.625502][ T423] netlink_sendmsg+0x735/0xc60 [ 36.625597][ T423] ____sys_sendmsg+0x419/0x850 [ 36.625693][ T423] ___sys_sendmsg+0x14e/0x1d0 [ 36.625787][ T423] __sys_sendmsg+0x145/0x1f0 [ 36.625884][ T423] do_syscall_64+0x117/0x590 [ 36.625984][ T423] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 36.626100][ T423] IN-SOFTIRQ-R at: [ 36.626159][ T423] __lock_acquire+0x378/0xc10 [ 36.626254][ T423] lock_acquire.part.0+0xbc/0x260 [ 36.626348][ T423] _raw_read_lock_bh+0x44/0x80 [ 36.626444][ T423] ipv6_get_lladdr+0xa8/0x3f0 [ 36.626539][ T423] ndisc_send_ns+0xf3/0x120 [ 36.626659][ T423] ndisc_solicit+0x314/0x480 [ 36.626756][ T423] neigh_probe+0xb3/0x100 [ 36.626858][ T423] __neigh_event_send+0x802/0x1070 [ 36.626958][ T423] neigh_resolve_output+0xbe/0x140 [ 36.627071][ T423] ip6_finish_output2+0x423/0x1300 [ 36.627168][ T423] ip6_finish_output+0x646/0xda0 [ 36.627264][ T423] ip6_output+0x23f/0x7f0 [ 36.627365][ T423] ip6_forward+0x10f4/0x2bd0 [ 36.627466][ T423] ipv6_rcv+0x3c1/0x5a0 [ 36.627565][ T423] __netif_receive_skb_one_core+0xfc/0x180 [ 36.627684][ T423] netif_receive_skb_internal+0xb0/0x330 [ 36.627801][ T423] netif_receive_skb+0x56/0x380 [ 36.627896][ T423] br_handle_frame_finish+0xc3b/0x1fb0 [bridge] [ 36.628031][ T423] br_handle_frame+0x739/0xf40 [bridge] [ 36.628157][ T423] __netif_receive_skb_core.constprop.0+0x536/0x2bc0 [ 36.628273][ T423] __netif_receive_skb_one_core+0xa2/0x180 [ 36.628388][ T423] process_backlog+0x2bc/0x1490 [ 36.628483][ T423] __napi_poll+0xa7/0x3b0 [ 36.628580][ T423] net_rx_action+0x513/0xf50 [ 36.628686][ T423] handle_softirqs+0x1d8/0x940 [ 36.628785][ T423] do_softirq+0xa9/0xe0 [ 36.628883][ T423] __local_bh_enable_ip+0x113/0x140 [ 36.628980][ T423] __neigh_event_send+0x30a/0x1070 [ 36.629084][ T423] neigh_resolve_output+0xbe/0x140 [ 36.629184][ T423] ip6_finish_output2+0x423/0x1300 [ 36.629285][ T423] ip6_finish_output+0x646/0xda0 [ 36.629380][ T423] ip6_output+0x23f/0x7f0 [ 36.629477][ T423] ip6_send_skb+0xee/0x350 [ 36.629573][ T423] rawv6_push_pending_frames+0x4cb/0xa90 [ 36.629690][ T423] rawv6_sendmsg+0x1434/0x1a50 [ 36.629785][ T423] __sys_sendto+0x2c9/0x400 [ 36.629882][ T423] __x64_sys_sendto+0xe4/0x1f0 [ 36.629978][ T423] do_syscall_64+0x117/0x590 [ 36.630077][ T423] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 36.630192][ T423] INITIAL USE at: [ 36.630253][ T423] __lock_acquire+0x378/0xc10 [ 36.630353][ T423] lock_acquire.part.0+0xbc/0x260 [ 36.630453][ T423] _raw_write_lock_bh+0x38/0x50 [ 36.630550][ T423] addrconf_permanent_addr+0x108/0x9c0 [ 36.630666][ T423] addrconf_notify+0x151/0xf30 [ 36.630769][ T423] notifier_call_chain+0xb0/0x320 [ 36.630866][ T423] __dev_notify_flags+0xde/0x280 [ 36.630966][ T423] netif_change_flags+0xfe/0x190 [ 36.631075][ T423] do_setlink.isra.0+0x1cc3/0x2750 [ 36.631180][ T423] rtnl_newlink+0x8d1/0xef0 [ 36.631280][ T423] rtnetlink_rcv_msg+0x6fd/0xbd0 [ 36.631375][ T423] netlink_rcv_skb+0x14e/0x3a0 [ 36.631469][ T423] netlink_unicast+0x486/0x750 [ 36.631563][ T423] netlink_sendmsg+0x735/0xc60 [ 36.631659][ T423] ____sys_sendmsg+0x419/0x850 [ 36.631763][ T423] ___sys_sendmsg+0x14e/0x1d0 [ 36.631860][ T423] __sys_sendmsg+0x145/0x1f0 [ 36.631955][ T423] do_syscall_64+0x117/0x590 [ 36.632055][ T423] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 36.632175][ T423] INITIAL READ USE at: [ 36.632254][ T423] __lock_acquire+0x378/0xc10 [ 36.632350][ T423] lock_acquire.part.0+0xbc/0x260 [ 36.632476][ T423] _raw_read_lock_bh+0x44/0x80 [ 36.632573][ T423] inet6_fill_ifla6_attrs+0x1aee/0x21e0 [ 36.632688][ T423] inet6_fill_link_af+0x5c/0xe0 [ 36.632788][ T423] rtnl_fill_ifinfo.isra.0+0x1a45/0x2c20 [ 36.632912][ T423] rtnl_getlink+0x9c9/0xeb0 [ 36.633012][ T423] rtnetlink_rcv_msg+0x6fd/0xbd0 [ 36.633108][ T423] netlink_rcv_skb+0x14e/0x3a0 [ 36.633204][ T423] netlink_unicast+0x486/0x750 [ 36.633302][ T423] netlink_sendmsg+0x735/0xc60 [ 36.633398][ T423] ____sys_sendmsg+0x419/0x850 [ 36.633495][ T423] ___sys_sendmsg+0x14e/0x1d0 [ 36.633592][ T423] __sys_sendmsg+0x145/0x1f0 [ 36.633687][ T423] do_syscall_64+0x117/0x590 [ 36.633783][ T423] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 36.633900][ T423] } [ 36.633940][ T423] ... key at: [] __key.37+0x0/0x40 [ 36.634064][ T423] ... acquired at: [ 36.634123][ T423] __lock_acquire+0x508/0xc10 [ 36.634201][ T423] lock_acquire.part.0+0xbc/0x260 [ 36.634277][ T423] _raw_spin_lock+0x33/0x40 [ 36.634352][ T423] ipv6_add_acaddr_hash.isra.0+0x16d/0x350 [ 36.634489][ T423] __ipv6_dev_ac_inc+0x57e/0x930 [ 36.634567][ T423] addrconf_join_anycast+0x1bc/0x260 [ 36.634642][ T423] __ipv6_ifa_notify+0x75b/0xad0 [ 36.634717][ T423] addrconf_dad_completed+0x14b/0xe10 [ 36.634836][ T423] addrconf_dad_work+0x3c1/0x930 [ 36.634912][ T423] process_one_work+0xdf8/0x1410 [ 36.634989][ T423] worker_thread+0x4f1/0xd60 [ 36.635071][ T423] kthread+0x367/0x460 [ 36.635170][ T423] ret_from_fork+0x474/0x6b0 [ 36.635249][ T423] ret_from_fork_asm+0x11/0x20 [ 36.635327][ T423] [ 36.635370][ T423] -> (acaddr_hash_lock){+.+.}-{3:3} { [ 36.635449][ T423] HARDIRQ-ON-W at: [ 36.635545][ T423] __lock_acquire+0x378/0xc10 [ 36.635641][ T423] lock_acquire.part.0+0xbc/0x260 [ 36.635738][ T423] _raw_spin_lock+0x33/0x40 [ 36.635877][ T423] ipv6_add_acaddr_hash.isra.0+0x16d/0x350 [ 36.635993][ T423] __ipv6_dev_ac_inc+0x57e/0x930 [ 36.636090][ T423] addrconf_join_anycast+0x1bc/0x260 [ 36.636228][ T423] __ipv6_ifa_notify+0x75b/0xad0 [ 36.636325][ T423] addrconf_dad_completed+0x14b/0xe10 [ 36.636426][ T423] addrconf_dad_work+0x3c1/0x930 [ 36.636564][ T423] process_one_work+0xdf8/0x1410 [ 36.636673][ T423] worker_thread+0x4f1/0xd60 [ 36.636770][ T423] kthread+0x367/0x460 [ 36.636847][ T423] ret_from_fork+0x474/0x6b0 [ 36.636990][ T423] ret_from_fork_asm+0x11/0x20 [ 36.637090][ T423] SOFTIRQ-ON-W at: [ 36.637148][ T423] __lock_acquire+0x378/0xc10 [ 36.637284][ T423] lock_acquire.part.0+0xbc/0x260 [ 36.637381][ T423] _raw_spin_lock+0x33/0x40 [ 36.637477][ T423] __ipv6_dev_ac_dec+0x236/0x5d0 [ 36.637573][ T423] addrconf_leave_anycast+0x1bc/0x260 [ 36.637710][ T423] dev_forward_change+0x3b3/0x850 [ 36.637808][ T423] addrconf_fixup_forwarding+0x2af/0x4d0 [ 36.637924][ T423] addrconf_sysctl_forward+0x209/0x2b0 [ 36.638064][ T423] proc_sys_call_handler+0x31b/0x480 [ 36.638162][ T423] new_sync_write+0x333/0x750 [ 36.638258][ T423] vfs_write+0x6a4/0xc10 [ 36.638397][ T423] ksys_write+0x116/0x250 [ 36.638492][ T423] do_syscall_64+0x117/0x590 [ 36.638600][ T423] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 36.638756][ T423] INITIAL USE at: [ 36.638818][ T423] __lock_acquire+0x378/0xc10 [ 36.638914][ T423] lock_acquire.part.0+0xbc/0x260 [ 36.639055][ T423] _raw_spin_lock+0x33/0x40 [ 36.639152][ T423] ipv6_add_acaddr_hash.isra.0+0x16d/0x350 [ 36.639267][ T423] __ipv6_dev_ac_inc+0x57e/0x930 [ 36.639409][ T423] addrconf_join_anycast+0x1bc/0x260 [ 36.639505][ T423] __ipv6_ifa_notify+0x75b/0xad0 [ 36.639602][ T423] addrconf_dad_completed+0x14b/0xe10 [ 36.639699][ T423] addrconf_dad_work+0x3c1/0x930 [ 36.639843][ T423] process_one_work+0xdf8/0x1410 [ 36.639941][ T423] worker_thread+0x4f1/0xd60 [ 36.640042][ T423] kthread+0x367/0x460 [ 36.640165][ T423] ret_from_fork+0x474/0x6b0 [ 36.640261][ T423] ret_from_fork_asm+0x11/0x20 [ 36.640357][ T423] } [ 36.640395][ T423] ... key at: [] acaddr_hash_lock+0x18/0x45a0 [ 36.640553][ T423] ... acquired at: [ 36.640610][ T423] mark_lock+0x1d7/0xa00 [ 36.640695][ T423] mark_usage+0x105/0x170 [ 36.640770][ T423] __lock_acquire+0x378/0xc10 [ 36.640886][ T423] lock_acquire.part.0+0xbc/0x260 [ 36.640963][ T423] _raw_spin_lock+0x33/0x40 [ 36.641046][ T423] __ipv6_dev_ac_dec+0x236/0x5d0 [ 36.641121][ T423] addrconf_leave_anycast+0x1bc/0x260 [ 36.641240][ T423] dev_forward_change+0x3b3/0x850 [ 36.641317][ T423] addrconf_fixup_forwarding+0x2af/0x4d0 [ 36.641414][ T423] addrconf_sysctl_forward+0x209/0x2b0 [ 36.641529][ T423] proc_sys_call_handler+0x31b/0x480 [ 36.641605][ T423] new_sync_write+0x333/0x750 [ 36.641681][ T423] vfs_write+0x6a4/0xc10 [ 36.641764][ T423] ksys_write+0x116/0x250 [ 36.641882][ T423] do_syscall_64+0x117/0x590 [ 36.641957][ T423] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 36.642056][ T423] [ 36.642094][ T423] [ 36.642094][ T423] stack backtrace: [ 36.642231][ T423] CPU: 3 UID: 0 PID: 423 Comm: sysctl Not tainted 7.1.0-rc5-virtme #1 PREEMPT(full) [ 36.642234][ T423] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 36.642236][ T423] Call Trace: [ 36.642237][ T423] [ 36.642238][ T423] dump_stack_lvl+0x6f/0xa0 [ 36.642243][ T423] print_irq_inversion_bug.part.0.cold+0xe6/0x143 [ 36.642246][ T423] mark_lock_irq+0x989/0x9c0 [ 36.642249][ T423] mark_lock+0x1d7/0xa00 [ 36.642250][ T423] mark_usage+0x105/0x170 [ 36.642252][ T423] __lock_acquire+0x378/0xc10 [ 36.642254][ T423] lock_acquire.part.0+0xbc/0x260 [ 36.642255][ T423] ? __ipv6_dev_ac_dec+0x236/0x5d0 [ 36.642257][ T423] ? rcu_is_watching+0x15/0xd0 [ 36.642260][ T423] ? do_raw_read_unlock+0x70/0x70 [ 36.642262][ T423] ? lock_acquire+0x134/0x160 [ 36.642264][ T423] _raw_spin_lock+0x33/0x40 [ 36.642265][ T423] ? __ipv6_dev_ac_dec+0x236/0x5d0 [ 36.642266][ T423] __ipv6_dev_ac_dec+0x236/0x5d0 [ 36.642268][ T423] addrconf_leave_anycast+0x1bc/0x260 [ 36.642270][ T423] ? find_held_lock+0x2b/0x80 [ 36.642273][ T423] ? __ipv6_isatap_ifid+0x210/0x210 [ 36.642275][ T423] ? mark_held_locks+0x40/0x70 [ 36.642276][ T423] ? lockdep_hardirqs_on+0x8c/0x130 [ 36.642279][ T423] dev_forward_change+0x3b3/0x850 [ 36.642281][ T423] ? addrconf_fixup_forwarding+0x4d/0x4d0 [ 36.642283][ T423] ? addrconf_sysctl_proxy_ndp+0x2f0/0x2f0 [ 36.642285][ T423] addrconf_fixup_forwarding+0x2af/0x4d0 [ 36.642288][ T423] addrconf_sysctl_forward+0x209/0x2b0 [ 36.642290][ T423] ? addrconf_fixup_forwarding+0x4d0/0x4d0 [ 36.642292][ T423] ? addrconf_fixup_forwarding+0x4d0/0x4d0 [ 36.642294][ T423] ? __kvmalloc_node_noprof+0x305/0x8a0 [ 36.642297][ T423] ? proc_sys_call_handler+0x25d/0x480 [ 36.642300][ T423] proc_sys_call_handler+0x31b/0x480 [ 36.642302][ T423] ? proc_sys_lookup+0x3d0/0x3d0 [ 36.642304][ T423] ? rcu_read_unlock+0x1b/0x70 [ 36.642306][ T423] ? do_raw_spin_unlock+0x59/0x250 [ 36.642308][ T423] ? rcu_lockdep_current_cpu_online+0x39/0x1b0 [ 36.642310][ T423] ? rcu_read_lock_any_held+0x3c/0x90 [ 36.642311][ T423] ? proc_sys_call_handler+0x480/0x480 [ 36.642313][ T423] new_sync_write+0x333/0x750 [ 36.642315][ T423] ? __lock_acquire+0x508/0xc10 [ 36.642316][ T423] ? new_sync_read+0x740/0x740 [ 36.642318][ T423] ? lock_acquire.part.0+0xbc/0x260 [ 36.642319][ T423] ? ksys_write+0x116/0x250 [ 36.642321][ T423] vfs_write+0x6a4/0xc10 [ 36.642323][ T423] ksys_write+0x116/0x250 [ 36.642324][ T423] ? __ia32_sys_read+0xc0/0xc0 [ 36.642326][ T423] ? rcu_is_watching+0x15/0xd0 [ 36.642327][ T423] ? rcu_is_watching+0x15/0xd0 [ 36.642329][ T423] do_syscall_64+0x117/0x590 [ 36.642330][ T423] ? trace_hardirqs_off+0xd/0x30 [ 36.642333][ T423] ? exc_page_fault+0xee/0x100 [ 36.642335][ T423] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 36.642337][ T423] RIP: 0033:0x7fde8dbd808e [ 36.642339][ T423] Code: 4d 89 d8 e8 94 bd 00 00 4c 8b 5d f8 41 8b 93 08 03 00 00 59 5e 48 83 f8 fc 74 11 c9 c3 0f 1f 80 00 00 00 00 48 8b 45 10 0f 05 c3 83 e2 39 83 fa 08 75 e7 e8 03 ff ff ff 0f 1f 00 f3 0f 1e fa [ 36.642341][ T423] RSP: 002b:00007ffcde8f6480 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 36.642344][ T423] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fde8dbd808e [ 36.642345][ T423] RDX: 0000000000000002 RSI: 000055f224d274b0 RDI: 0000000000000005 [ 36.642346][ T423] RBP: 00007ffcde8f6490 R08: 0000000000000000 R09: 0000000000000000 [ 36.642347][ T423] R10: 0000000000000000 R11: 0000000000000202 R12: 000055f224d295e0 [ 36.642348][ T423] R13: 000055f224d27470 R14: 0000000000000002 R15: 0000000000000000 [ 36.642350][ T423] [ 36.798384][ T434] br1: port 1(veth1) entered disabled state [ 36.814927][ T435] veth1: left allmulticast mode [ 36.815079][ T435] veth1: left promiscuous mode [ 36.815274][ T435] br1: port 1(veth1) entered disabled state