[ 53.282065][ T628] 8021q: 802.1Q VLAN Support v1.8 [ 54.074272][ T664] br1: port 1(veth1) entered blocking state [ 54.074544][ T664] br1: port 1(veth1) entered disabled state [ 54.074777][ T664] veth1: entered allmulticast mode [ 54.076223][ T664] veth1: entered promiscuous mode [ 54.106683][ T70] br1: port 1(veth1) entered blocking state [ 54.106891][ T70] br1: port 1(veth1) entered forwarding state [ 80.646110][ T836] [ 80.646197][ T836] ======================================================== [ 80.646325][ T836] WARNING: possible irq lock inversion dependency detected [ 80.646457][ T836] 7.1.0-rc5-virtme #1 Not tainted [ 80.646542][ T836] -------------------------------------------------------- [ 80.646663][ T836] sysctl/836 just changed the state of lock: [ 80.646764][ T836] ffffffffa217e8b8 (acaddr_hash_lock){+.+.}-{3:3}, at: __ipv6_dev_ac_dec+0x236/0x5d0 [ 80.646915][ T836] but this lock was taken by another, SOFTIRQ-safe lock in the past: [ 80.647036][ T836] (&ndev->lock){++--}-{3:3} [ 80.647038][ T836] [ 80.647038][ T836] [ 80.647038][ T836] and interrupts could create inverse lock ordering between them. [ 80.647038][ T836] [ 80.647340][ T836] [ 80.647340][ T836] other info that might help us debug this: [ 80.647472][ T836] Possible interrupt unsafe locking scenario: [ 80.647472][ T836] [ 80.647599][ T836] CPU0 CPU1 [ 80.647686][ T836] ---- ---- [ 80.647769][ T836] lock(acaddr_hash_lock); [ 80.647855][ T836] local_irq_disable(); [ 80.647958][ T836] lock(&ndev->lock); [ 80.648063][ T836] lock(acaddr_hash_lock); [ 80.648165][ T836] [ 80.648225][ T836] lock(&ndev->lock); [ 80.648289][ T836] [ 80.648289][ T836] *** DEADLOCK *** [ 80.648289][ T836] [ 80.648416][ T836] 2 locks held by sysctl/836: [ 80.648504][ T836] #0: ff1100000a0243e0 (sb_writers#3){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 80.648648][ T836] #1: ffffffffa208c9c0 (rtnl_mutex){+.+.}-{4:4}, at: addrconf_fixup_forwarding+0x4d/0x4d0 [ 80.648817][ T836] [ 80.648817][ T836] the shortest dependencies between 2nd lock and 1st lock: [ 80.648957][ T836] -> (&ndev->lock){++--}-{3:3} { [ 80.649042][ T836] HARDIRQ-ON-W at: [ 80.649105][ T836] __lock_acquire+0x378/0xc10 [ 80.649207][ T836] lock_acquire.part.0+0xbc/0x260 [ 80.649307][ T836] _raw_write_lock_bh+0x38/0x50 [ 80.649412][ T836] addrconf_permanent_addr+0x108/0x9c0 [ 80.649532][ T836] addrconf_notify+0x151/0xf30 [ 80.649632][ T836] notifier_call_chain+0xb0/0x320 [ 80.649731][ T836] __dev_notify_flags+0xde/0x280 [ 80.649832][ T836] netif_change_flags+0xfe/0x190 [ 80.649932][ T836] do_setlink.isra.0+0x1cc3/0x2750 [ 80.650035][ T836] rtnl_newlink+0x8d1/0xef0 [ 80.650141][ T836] rtnetlink_rcv_msg+0x6fd/0xbd0 [ 80.650242][ T836] netlink_rcv_skb+0x14e/0x3a0 [ 80.650344][ T836] netlink_unicast+0x486/0x750 [ 80.650450][ T836] netlink_sendmsg+0x735/0xc60 [ 80.650553][ T836] ____sys_sendmsg+0x419/0x850 [ 80.650658][ T836] ___sys_sendmsg+0x14e/0x1d0 [ 80.650765][ T836] __sys_sendmsg+0x145/0x1f0 [ 80.650867][ T836] do_syscall_64+0x117/0x590 [ 80.650969][ T836] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 80.651093][ T836] HARDIRQ-ON-R at: [ 80.651157][ T836] __lock_acquire+0x378/0xc10 [ 80.651263][ T836] lock_acquire.part.0+0xbc/0x260 [ 80.651370][ T836] _raw_read_lock_bh+0x44/0x80 [ 80.651477][ T836] inet6_fill_ifla6_attrs+0x1aee/0x21e0 [ 80.651602][ T836] inet6_fill_link_af+0x5c/0xe0 [ 80.651709][ T836] rtnl_fill_ifinfo.isra.0+0x1a45/0x2c20 [ 80.651835][ T836] rtnl_getlink+0x9c9/0xeb0 [ 80.651978][ T836] rtnetlink_rcv_msg+0x6fd/0xbd0 [ 80.652078][ T836] netlink_rcv_skb+0x14e/0x3a0 [ 80.652180][ T836] netlink_unicast+0x486/0x750 [ 80.652321][ T836] netlink_sendmsg+0x735/0xc60 [ 80.652427][ T836] ____sys_sendmsg+0x419/0x850 [ 80.652528][ T836] ___sys_sendmsg+0x14e/0x1d0 [ 80.652670][ T836] __sys_sendmsg+0x145/0x1f0 [ 80.652771][ T836] do_syscall_64+0x117/0x590 [ 80.652870][ T836] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 80.653029][ T836] IN-SOFTIRQ-W at: [ 80.653088][ T836] __lock_acquire+0x378/0xc10 [ 80.653185][ T836] lock_acquire.part.0+0xbc/0x260 [ 80.653288][ T836] _raw_write_lock+0x33/0x40 [ 80.653438][ T836] addrconf_rs_timer+0xb0/0x770 [ 80.653537][ T836] call_timer_fn+0x163/0x4f0 [ 80.653642][ T836] __run_timers+0x68f/0xab0 [ 80.653781][ T836] run_timer_softirq+0xf0/0x160 [ 80.653888][ T836] handle_softirqs+0x1d8/0x940 [ 80.653989][ T836] __irq_exit_rcu+0x103/0x1c0 [ 80.654128][ T836] irq_exit_rcu+0xe/0x30 [ 80.654226][ T836] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 80.654350][ T836] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 80.654522][ T836] pv_native_safe_halt+0xf/0x10 [ 80.654625][ T836] default_idle+0x9/0x10 [ 80.654724][ T836] default_idle_call+0x6e/0xb0 [ 80.654869][ T836] cpuidle_idle_call.constprop.0+0x237/0x410 [ 80.654989][ T836] do_idle+0xf5/0x160 [ 80.655088][ T836] cpu_startup_entry+0x53/0x70 [ 80.655228][ T836] start_secondary+0x204/0x2b0 [ 80.655340][ T836] common_startup_64+0x13e/0x148 [ 80.655445][ T836] IN-SOFTIRQ-R at: [ 80.655505][ T836] __lock_acquire+0x378/0xc10 [ 80.655648][ T836] lock_acquire.part.0+0xbc/0x260 [ 80.655751][ T836] _raw_read_lock_bh+0x44/0x80 [ 80.655852][ T836] ipv6_get_lladdr+0xa8/0x3f0 [ 80.655996][ T836] addrconf_rs_timer+0x251/0x770 [ 80.656100][ T836] call_timer_fn+0x163/0x4f0 [ 80.656198][ T836] __run_timers+0x68f/0xab0 [ 80.656337][ T836] run_timer_softirq+0xf0/0x160 [ 80.656447][ T836] handle_softirqs+0x1d8/0x940 [ 80.656553][ T836] __irq_exit_rcu+0x103/0x1c0 [ 80.656696][ T836] irq_exit_rcu+0xe/0x30 [ 80.656795][ T836] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 80.656917][ T836] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 80.657075][ T836] pv_native_safe_halt+0xf/0x10 [ 80.657176][ T836] default_idle+0x9/0x10 [ 80.657281][ T836] default_idle_call+0x6e/0xb0 [ 80.657425][ T836] cpuidle_idle_call.constprop.0+0x237/0x410 [ 80.657548][ T836] do_idle+0xf5/0x160 [ 80.657651][ T836] cpu_startup_entry+0x53/0x70 [ 80.657794][ T836] start_secondary+0x204/0x2b0 [ 80.657897][ T836] common_startup_64+0x13e/0x148 [ 80.657996][ T836] INITIAL USE at: [ 80.658055][ T836] __lock_acquire+0x378/0xc10 [ 80.658205][ T836] lock_acquire.part.0+0xbc/0x260 [ 80.658309][ T836] _raw_write_lock_bh+0x38/0x50 [ 80.658414][ T836] addrconf_permanent_addr+0x108/0x9c0 [ 80.658581][ T836] addrconf_notify+0x151/0xf30 [ 80.658685][ T836] notifier_call_chain+0xb0/0x320 [ 80.658784][ T836] __dev_notify_flags+0xde/0x280 [ 80.658924][ T836] netif_change_flags+0xfe/0x190 [ 80.659025][ T836] do_setlink.isra.0+0x1cc3/0x2750 [ 80.659124][ T836] rtnl_newlink+0x8d1/0xef0 [ 80.659264][ T836] rtnetlink_rcv_msg+0x6fd/0xbd0 [ 80.659368][ T836] netlink_rcv_skb+0x14e/0x3a0 [ 80.659467][ T836] netlink_unicast+0x486/0x750 [ 80.659569][ T836] netlink_sendmsg+0x735/0xc60 [ 80.659712][ T836] ____sys_sendmsg+0x419/0x850 [ 80.659810][ T836] ___sys_sendmsg+0x14e/0x1d0 [ 80.659913][ T836] __sys_sendmsg+0x145/0x1f0 [ 80.660060][ T836] do_syscall_64+0x117/0x590 [ 80.660165][ T836] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 80.660291][ T836] INITIAL READ USE at: [ 80.660419][ T836] __lock_acquire+0x378/0xc10 [ 80.660518][ T836] lock_acquire.part.0+0xbc/0x260 [ 80.660637][ T836] _raw_read_lock_bh+0x44/0x80 [ 80.660781][ T836] inet6_fill_ifla6_attrs+0x1aee/0x21e0 [ 80.660902][ T836] inet6_fill_link_af+0x5c/0xe0 [ 80.661001][ T836] rtnl_fill_ifinfo.isra.0+0x1a45/0x2c20 [ 80.661169][ T836] rtnl_getlink+0x9c9/0xeb0 [ 80.661268][ T836] rtnetlink_rcv_msg+0x6fd/0xbd0 [ 80.661372][ T836] netlink_rcv_skb+0x14e/0x3a0 [ 80.661516][ T836] netlink_unicast+0x486/0x750 [ 80.661614][ T836] netlink_sendmsg+0x735/0xc60 [ 80.661714][ T836] ____sys_sendmsg+0x419/0x850 [ 80.661855][ T836] ___sys_sendmsg+0x14e/0x1d0 [ 80.661956][ T836] __sys_sendmsg+0x145/0x1f0 [ 80.662057][ T836] do_syscall_64+0x117/0x590 [ 80.662196][ T836] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 80.662323][ T836] } [ 80.662368][ T836] ... key at: [] __key.37+0x0/0x40 [ 80.662495][ T836] ... acquired at: [ 80.662595][ T836] __lock_acquire+0x508/0xc10 [ 80.662678][ T836] lock_acquire.part.0+0xbc/0x260 [ 80.662762][ T836] _raw_spin_lock+0x33/0x40 [ 80.662841][ T836] ipv6_add_acaddr_hash.isra.0+0x16d/0x350 [ 80.662985][ T836] __ipv6_dev_ac_inc+0x57e/0x930 [ 80.663070][ T836] addrconf_join_anycast+0x1bc/0x260 [ 80.663151][ T836] __ipv6_ifa_notify+0x75b/0xad0 [ 80.663232][ T836] addrconf_dad_completed+0x14b/0xe10 [ 80.663352][ T836] addrconf_dad_work+0x3c1/0x930 [ 80.663441][ T836] process_one_work+0xdf8/0x1410 [ 80.663522][ T836] worker_thread+0x4f1/0xd60 [ 80.663606][ T836] kthread+0x367/0x460 [ 80.663706][ T836] ret_from_fork+0x474/0x6b0 [ 80.663786][ T836] ret_from_fork_asm+0x11/0x20 [ 80.663873][ T836] [ 80.663916][ T836] -> (acaddr_hash_lock){+.+.}-{3:3} { [ 80.663996][ T836] HARDIRQ-ON-W at: [ 80.664100][ T836] __lock_acquire+0x378/0xc10 [ 80.664199][ T836] lock_acquire.part.0+0xbc/0x260 [ 80.664298][ T836] _raw_spin_lock+0x33/0x40 [ 80.664442][ T836] ipv6_add_acaddr_hash.isra.0+0x16d/0x350 [ 80.664564][ T836] __ipv6_dev_ac_inc+0x57e/0x930 [ 80.664670][ T836] addrconf_join_anycast+0x1bc/0x260 [ 80.664810][ T836] __ipv6_ifa_notify+0x75b/0xad0 [ 80.664913][ T836] addrconf_dad_completed+0x14b/0xe10 [ 80.665012][ T836] addrconf_dad_work+0x3c1/0x930 [ 80.665158][ T836] process_one_work+0xdf8/0x1410 [ 80.665261][ T836] worker_thread+0x4f1/0xd60 [ 80.665365][ T836] kthread+0x367/0x460 [ 80.665445][ T836] ret_from_fork+0x474/0x6b0 [ 80.665589][ T836] ret_from_fork_asm+0x11/0x20 [ 80.665693][ T836] SOFTIRQ-ON-W at: [ 80.665753][ T836] __lock_acquire+0x378/0xc10 [ 80.665893][ T836] lock_acquire.part.0+0xbc/0x260 [ 80.665998][ T836] _raw_spin_lock+0x33/0x40 [ 80.666102][ T836] __ipv6_dev_ac_dec+0x236/0x5d0 [ 80.666201][ T836] addrconf_leave_anycast+0x1bc/0x260 [ 80.666347][ T836] dev_forward_change+0x3b3/0x850 [ 80.666455][ T836] addrconf_fixup_forwarding+0x2af/0x4d0 [ 80.666578][ T836] addrconf_sysctl_forward+0x209/0x2b0 [ 80.666722][ T836] proc_sys_call_handler+0x31b/0x480 [ 80.666827][ T836] new_sync_write+0x333/0x750 [ 80.666928][ T836] vfs_write+0x6a4/0xc10 [ 80.667069][ T836] ksys_write+0x116/0x250 [ 80.667176][ T836] do_syscall_64+0x117/0x590 [ 80.667274][ T836] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 80.667441][ T836] INITIAL USE at: [ 80.667500][ T836] __lock_acquire+0x378/0xc10 [ 80.667606][ T836] lock_acquire.part.0+0xbc/0x260 [ 80.667751][ T836] _raw_spin_lock+0x33/0x40 [ 80.667855][ T836] ipv6_add_acaddr_hash.isra.0+0x16d/0x350 [ 80.667972][ T836] __ipv6_dev_ac_inc+0x57e/0x930 [ 80.668111][ T836] addrconf_join_anycast+0x1bc/0x260 [ 80.668211][ T836] __ipv6_ifa_notify+0x75b/0xad0 [ 80.668312][ T836] addrconf_dad_completed+0x14b/0xe10 [ 80.668418][ T836] addrconf_dad_work+0x3c1/0x930 [ 80.668564][ T836] process_one_work+0xdf8/0x1410 [ 80.668663][ T836] worker_thread+0x4f1/0xd60 [ 80.668761][ T836] kthread+0x367/0x460 [ 80.668883][ T836] ret_from_fork+0x474/0x6b0 [ 80.668982][ T836] ret_from_fork_asm+0x11/0x20 [ 80.669080][ T836] } [ 80.669120][ T836] ... key at: [] acaddr_hash_lock+0x18/0x45a0 [ 80.669278][ T836] ... acquired at: [ 80.669336][ T836] mark_lock+0x1d7/0xa00 [ 80.669424][ T836] mark_usage+0x105/0x170 [ 80.669501][ T836] __lock_acquire+0x378/0xc10 [ 80.669621][ T836] lock_acquire.part.0+0xbc/0x260 [ 80.669704][ T836] _raw_spin_lock+0x33/0x40 [ 80.669781][ T836] __ipv6_dev_ac_dec+0x236/0x5d0 [ 80.669858][ T836] addrconf_leave_anycast+0x1bc/0x260 [ 80.669976][ T836] dev_forward_change+0x3b3/0x850 [ 80.670055][ T836] addrconf_fixup_forwarding+0x2af/0x4d0 [ 80.670153][ T836] addrconf_sysctl_forward+0x209/0x2b0 [ 80.670274][ T836] proc_sys_call_handler+0x31b/0x480 [ 80.670353][ T836] new_sync_write+0x333/0x750 [ 80.670438][ T836] vfs_write+0x6a4/0xc10 [ 80.670517][ T836] ksys_write+0x116/0x250 [ 80.670640][ T836] do_syscall_64+0x117/0x590 [ 80.670723][ T836] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 80.670820][ T836] [ 80.670859][ T836] [ 80.670859][ T836] stack backtrace: [ 80.670998][ T836] CPU: 3 UID: 0 PID: 836 Comm: sysctl Not tainted 7.1.0-rc5-virtme #1 PREEMPT(full) [ 80.671001][ T836] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 80.671003][ T836] Call Trace: [ 80.671004][ T836] [ 80.671005][ T836] dump_stack_lvl+0x6f/0xa0 [ 80.671009][ T836] print_irq_inversion_bug.part.0.cold+0xe6/0x143 [ 80.671012][ T836] mark_lock_irq+0x989/0x9c0 [ 80.671015][ T836] mark_lock+0x1d7/0xa00 [ 80.671017][ T836] mark_usage+0x105/0x170 [ 80.671018][ T836] __lock_acquire+0x378/0xc10 [ 80.671020][ T836] lock_acquire.part.0+0xbc/0x260 [ 80.671022][ T836] ? __ipv6_dev_ac_dec+0x236/0x5d0 [ 80.671023][ T836] ? rcu_is_watching+0x15/0xd0 [ 80.671026][ T836] ? do_raw_read_unlock+0x70/0x70 [ 80.671028][ T836] ? lock_acquire+0x134/0x160 [ 80.671030][ T836] _raw_spin_lock+0x33/0x40 [ 80.671031][ T836] ? __ipv6_dev_ac_dec+0x236/0x5d0 [ 80.671032][ T836] __ipv6_dev_ac_dec+0x236/0x5d0 [ 80.671034][ T836] addrconf_leave_anycast+0x1bc/0x260 [ 80.671036][ T836] ? find_held_lock+0x2b/0x80 [ 80.671039][ T836] ? __ipv6_isatap_ifid+0x210/0x210 [ 80.671040][ T836] ? mark_held_locks+0x40/0x70 [ 80.671042][ T836] ? lockdep_hardirqs_on+0x8c/0x130 [ 80.671044][ T836] dev_forward_change+0x3b3/0x850 [ 80.671046][ T836] ? addrconf_fixup_forwarding+0x4d/0x4d0 [ 80.671048][ T836] ? addrconf_sysctl_proxy_ndp+0x2f0/0x2f0 [ 80.671051][ T836] addrconf_fixup_forwarding+0x2af/0x4d0 [ 80.671053][ T836] addrconf_sysctl_forward+0x209/0x2b0 [ 80.671055][ T836] ? addrconf_fixup_forwarding+0x4d0/0x4d0 [ 80.671057][ T836] ? addrconf_fixup_forwarding+0x4d0/0x4d0 [ 80.671059][ T836] ? __kvmalloc_node_noprof+0x305/0x8a0 [ 80.671062][ T836] ? proc_sys_call_handler+0x25d/0x480 [ 80.671065][ T836] proc_sys_call_handler+0x31b/0x480 [ 80.671067][ T836] ? proc_sys_lookup+0x3d0/0x3d0 [ 80.671069][ T836] ? rcu_read_unlock+0x1b/0x70 [ 80.671071][ T836] ? do_raw_spin_unlock+0x59/0x250 [ 80.671072][ T836] ? rcu_lockdep_current_cpu_online+0x39/0x1b0 [ 80.671074][ T836] ? rcu_read_lock_any_held+0x3c/0x90 [ 80.671076][ T836] ? proc_sys_call_handler+0x480/0x480 [ 80.671078][ T836] new_sync_write+0x333/0x750 [ 80.671080][ T836] ? __lock_acquire+0x508/0xc10 [ 80.671081][ T836] ? new_sync_read+0x740/0x740 [ 80.671083][ T836] ? lock_acquire.part.0+0xbc/0x260 [ 80.671084][ T836] ? ksys_write+0x116/0x250 [ 80.671086][ T836] vfs_write+0x6a4/0xc10 [ 80.671088][ T836] ksys_write+0x116/0x250 [ 80.671089][ T836] ? __ia32_sys_read+0xc0/0xc0 [ 80.671090][ T836] ? rcu_is_watching+0x15/0xd0 [ 80.671092][ T836] ? rcu_is_watching+0x15/0xd0 [ 80.671094][ T836] do_syscall_64+0x117/0x590 [ 80.671095][ T836] ? trace_hardirqs_off+0xd/0x30 [ 80.671098][ T836] ? exc_page_fault+0xee/0x100 [ 80.671100][ T836] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 80.671101][ T836] RIP: 0033:0x7fb45782108e [ 80.671104][ T836] Code: 4d 89 d8 e8 94 bd 00 00 4c 8b 5d f8 41 8b 93 08 03 00 00 59 5e 48 83 f8 fc 74 11 c9 c3 0f 1f 80 00 00 00 00 48 8b 45 10 0f 05 c3 83 e2 39 83 fa 08 75 e7 e8 03 ff ff ff 0f 1f 00 f3 0f 1e fa [ 80.671106][ T836] RSP: 002b:00007ffe98d7fe10 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 80.671109][ T836] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fb45782108e [ 80.671110][ T836] RDX: 0000000000000002 RSI: 00005565cf78c4b0 RDI: 0000000000000005 [ 80.671111][ T836] RBP: 00007ffe98d7fe20 R08: 0000000000000000 R09: 0000000000000000 [ 80.671112][ T836] R10: 0000000000000000 R11: 0000000000000202 R12: 00005565cf78e5e0 [ 80.671112][ T836] R13: 00005565cf78c470 R14: 0000000000000002 R15: 0000000000000000 [ 80.671115][ T836] [ 80.856392][ T847] br1: port 1(veth1) entered disabled state [ 80.874087][ T848] veth1: left allmulticast mode [ 80.874211][ T848] veth1: left promiscuous mode [ 80.874377][ T848] br1: port 1(veth1) entered disabled state