[ 10.050646][ T230] 8021q: 802.1Q VLAN Support v1.8 [ 11.447208][ T282] br1: port 1(vx10) entered blocking state [ 11.447477][ T282] br1: port 1(vx10) entered disabled state [ 11.447751][ T282] vx10: entered allmulticast mode [ 11.449239][ T282] vx10: entered promiscuous mode [ 11.449641][ T282] br1: port 1(vx10) entered blocking state [ 11.449868][ T282] br1: port 1(vx10) entered forwarding state [ 11.573945][ T287] br1: port 2(vx20) entered blocking state [ 11.574232][ T287] br1: port 2(vx20) entered disabled state [ 11.574421][ T287] vx20: entered allmulticast mode [ 11.576159][ T287] vx20: entered promiscuous mode [ 11.576529][ T287] br1: port 2(vx20) entered blocking state [ 11.576783][ T287] br1: port 2(vx20) entered forwarding state [ 11.636923][ T289] br1: port 3(veth1) entered blocking state [ 11.637148][ T289] br1: port 3(veth1) entered disabled state [ 11.637383][ T289] veth1: entered allmulticast mode [ 11.647236][ T289] veth1: entered promiscuous mode [ 11.671120][ T188] br1: port 3(veth1) entered blocking state [ 11.671378][ T188] br1: port 3(veth1) entered forwarding state [ 11.781401][ T294] br1: port 4(veth2) entered blocking state [ 11.781596][ T294] br1: port 4(veth2) entered disabled state [ 11.781795][ T294] veth2: entered allmulticast mode [ 11.783349][ T294] veth2: entered promiscuous mode [ 11.806985][ T38] br1: port 4(veth2) entered blocking state [ 11.807166][ T38] br1: port 4(veth2) entered forwarding state [ 13.019687][ T343] br2: port 1(w1) entered blocking state [ 13.019862][ T343] br2: port 1(w1) entered disabled state [ 13.020014][ T343] w1: entered allmulticast mode [ 13.021463][ T343] w1: entered promiscuous mode [ 13.235103][ T351] br2: port 2(vx10) entered blocking state [ 13.235304][ T351] br2: port 2(vx10) entered disabled state [ 13.235487][ T351] vx10: entered allmulticast mode [ 13.237007][ T351] vx10: entered promiscuous mode [ 13.237392][ T351] br2: port 2(vx10) entered blocking state [ 13.237567][ T351] br2: port 2(vx10) entered forwarding state [ 13.427758][ T358] br2: port 3(vx20) entered blocking state [ 13.427958][ T358] br2: port 3(vx20) entered disabled state [ 13.428139][ T358] vx20: entered allmulticast mode [ 13.429599][ T358] vx20: entered promiscuous mode [ 13.429953][ T358] br2: port 3(vx20) entered blocking state [ 13.430120][ T358] br2: port 3(vx20) entered forwarding state [ 13.660424][ T188] br2: port 1(w1) entered blocking state [ 13.660794][ T188] br2: port 1(w1) entered forwarding state [ 14.418875][ T399] br2: port 1(w1) entered blocking state [ 14.419046][ T399] br2: port 1(w1) entered disabled state [ 14.419199][ T399] w1: entered allmulticast mode [ 14.420641][ T399] w1: entered promiscuous mode [ 14.444555][ T400] br2: port 1(w1) entered blocking state [ 14.444767][ T400] br2: port 1(w1) entered forwarding state [ 14.637059][ T407] br2: port 2(vx10) entered blocking state [ 14.637317][ T407] br2: port 2(vx10) entered disabled state [ 14.637540][ T407] vx10: entered allmulticast mode [ 14.639589][ T407] vx10: entered promiscuous mode [ 14.640017][ T407] br2: port 2(vx10) entered blocking state [ 14.640261][ T407] br2: port 2(vx10) entered forwarding state [ 14.847549][ T414] br2: port 3(vx20) entered blocking state [ 14.847834][ T414] br2: port 3(vx20) entered disabled state [ 14.848095][ T414] vx20: entered allmulticast mode [ 14.849600][ T414] vx20: entered promiscuous mode [ 14.850109][ T414] br2: port 3(vx20) entered blocking state [ 14.850291][ T414] br2: port 3(vx20) entered forwarding state [ 23.367662][ T510] GACT probability NOT on [ 55.677032][ T579] jq (579) used greatest stack depth: 23920 bytes left [ 122.933421][ T1328] veth3: entered promiscuous mode [ 141.211387][ T2255] veth3: left promiscuous mode [ 141.376413][ T2261] veth3: entered promiscuous mode [ 159.658892][ T3188] veth3: left promiscuous mode [ 159.964690][ T3206] vx20: left allmulticast mode [ 159.964870][ T3206] vx20: left promiscuous mode [ 159.965107][ T3206] br1: port 2(vx20) entered disabled state [ 159.995577][ T3207] vx10: left allmulticast mode [ 159.995791][ T3207] vx10: left promiscuous mode [ 159.996070][ T3207] br1: port 1(vx10) entered disabled state [ 165.110573][ T3212] br1: port 1(vx10) entered blocking state [ 165.111359][ T3212] br1: port 1(vx10) entered disabled state [ 165.111564][ T3212] vx10: entered allmulticast mode [ 165.113057][ T3212] vx10: entered promiscuous mode [ 165.113396][ T3212] br1: port 1(vx10) entered blocking state [ 165.113569][ T3212] br1: port 1(vx10) entered forwarding state [ 165.174024][ T3214] br1: port 2(vx20) entered blocking state [ 165.174218][ T3214] br1: port 2(vx20) entered disabled state [ 165.174406][ T3214] vx20: entered allmulticast mode [ 165.176576][ T3214] vx20: entered promiscuous mode [ 165.176922][ T3214] br1: port 2(vx20) entered blocking state [ 165.177089][ T3214] br1: port 2(vx20) entered forwarding state [ 271.889990][ T4058] veth3: entered promiscuous mode [ 290.272162][ T4985] veth3: left promiscuous mode [ 290.446349][ T4991] veth3: entered promiscuous mode [ 308.797126][ T5918] veth3: left promiscuous mode [ 485.977120][ T69] vx20: left allmulticast mode [ 485.977367][ T69] vx20: left promiscuous mode [ 485.977708][ T69] br2: port 3(vx20) entered disabled state [ 485.981415][ T69] vx10: left allmulticast mode [ 485.981584][ T69] vx10: left promiscuous mode [ 485.981894][ T69] br2: port 2(vx10) entered disabled state [ 485.983188][ T69] w1: left allmulticast mode [ 485.983344][ T69] w1: left promiscuous mode [ 485.983563][ T69] br2: port 1(w1) entered disabled state [ 485.991275][ T69] [ 485.991346][ T69] ====================================================== [ 485.991468][ T69] WARNING: possible circular locking dependency detected [ 485.991584][ T69] 7.2.0-rc6-virtme #1 Not tainted [ 485.991682][ T69] ------------------------------------------------------ [ 485.991803][ T69] kworker/u16:1/69 is trying to acquire lock: [ 485.991923][ T69] ff1100000bf61108 ((&n->timer)){+.-.}-{0:0}, at: __timer_delete_sync+0xdb/0x130 [ 485.992098][ T69] [ 485.992098][ T69] but task is already holding lock: [ 485.992234][ T69] ff1100001764d358 (&tbl->lock){+.-.}-{3:3}, at: __neigh_ifdown.isra.0+0xa1/0xea0 [ 485.992401][ T69] [ 485.992401][ T69] which lock already depends on the new lock. [ 485.992401][ T69] [ 485.992593][ T69] [ 485.992593][ T69] the existing dependency chain (in reverse order) is: [ 485.992764][ T69] [ 485.992764][ T69] -> #1 (&tbl->lock){+.-.}-{3:3}: [ 485.992909][ T69] __lock_acquire+0x518/0xc20 [ 485.993010][ T69] lock_acquire.part.0+0xbc/0x260 [ 485.993103][ T69] _raw_spin_lock_bh+0x38/0x50 [ 485.993217][ T69] ___neigh_create+0x2be/0x1070 [ 485.993309][ T69] ip6_finish_output2+0xb64/0x1380 [ 485.993400][ T69] ip6_finish_output+0x646/0xda0 [ 485.993489][ T69] ip6_output+0x23f/0x7f0 [ 485.993580][ T69] ndisc_send_skb+0xba3/0x1520 [ 485.993681][ T69] ndisc_send_ns+0xa9/0x120 [ 485.993781][ T69] ndisc_solicit+0x314/0x480 [ 485.993878][ T69] neigh_probe+0xb3/0x100 [ 485.993971][ T69] neigh_timer_handler+0x191/0x1210 [ 485.994072][ T69] call_timer_fn+0x163/0x4c0 [ 485.994162][ T69] __run_timers+0x68f/0xaa0 [ 485.994256][ T69] run_timer_softirq+0xf0/0x160 [ 485.994344][ T69] handle_softirqs+0x1d8/0x8f0 [ 485.994436][ T69] __irq_exit_rcu+0x103/0x1c0 [ 485.994531][ T69] irq_exit_rcu+0xe/0x30 [ 485.994624][ T69] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 485.994743][ T69] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 485.994862][ T69] pv_native_safe_halt+0xf/0x10 [ 485.994953][ T69] default_idle+0x9/0x10 [ 485.995042][ T69] default_idle_call+0x6e/0xb0 [ 485.995130][ T69] cpuidle_idle_call.constprop.0+0x237/0x410 [ 485.995239][ T69] do_idle+0xd8/0x180 [ 485.995327][ T69] cpu_startup_entry+0x53/0x70 [ 485.995414][ T69] start_secondary+0x204/0x2b0 [ 485.995507][ T69] common_startup_64+0x13e/0x148 [ 485.995597][ T69] [ 485.995597][ T69] -> #0 ((&n->timer)){+.-.}-{0:0}: [ 485.995745][ T69] check_prev_add+0xd8/0xe90 [ 485.995835][ T69] validate_chain+0x6b8/0xc20 [ 485.995928][ T69] __lock_acquire+0x518/0xc20 [ 485.996021][ T69] lock_acquire.part.0+0xbc/0x260 [ 485.996115][ T69] __timer_delete_sync+0xea/0x130 [ 485.996208][ T69] neigh_flush_one+0x4bf/0x650 [ 485.996300][ T69] __neigh_ifdown.isra.0+0x1b5/0xea0 [ 485.996408][ T69] neigh_ifdown+0x10/0x20 [ 485.996496][ T69] rt6_disable_ip+0x183/0x200 [ 485.996585][ T69] addrconf_ifdown.isra.0+0x103/0x1670 [ 485.996703][ T69] addrconf_notify+0x2c8/0xf30 [ 485.996791][ T69] notifier_call_chain+0xae/0x300 [ 485.996909][ T69] netif_close_many+0x366/0x710 [ 485.997002][ T69] unregister_netdevice_many_notify+0x671/0x19a0 [ 485.997115][ T69] ops_undo_list+0x615/0x8f0 [ 485.997214][ T69] cleanup_net+0x431/0x890 [ 485.997303][ T69] process_one_work+0xdf8/0x1410 [ 485.997394][ T69] worker_thread+0x4f1/0xd60 [ 485.997481][ T69] kthread+0x367/0x460 [ 485.997574][ T69] ret_from_fork+0x474/0x6b0 [ 485.997662][ T69] ret_from_fork_asm+0x11/0x20 [ 485.997760][ T69] [ 485.997760][ T69] other info that might help us debug this: [ 485.997760][ T69] [ 485.997944][ T69] Possible unsafe locking scenario: [ 485.997944][ T69] [ 485.998079][ T69] CPU0 CPU1 [ 485.998168][ T69] ---- ---- [ 485.998259][ T69] lock(&tbl->lock); [ 485.998331][ T69] lock((&n->timer)); [ 485.998445][ T69] lock(&tbl->lock); [ 485.998622][ T69] lock((&n->timer)); [ 485.998718][ T69] [ 485.998718][ T69] *** DEADLOCK *** [ 485.998718][ T69] [ 485.998853][ T69] 5 locks held by kworker/u16:1/69: [ 485.998948][ T69] #0: ff110000019d9d40 ((wq_completion)netns){+.+.}-{0:0}, at: process_one_work+0xd55/0x1410 [ 485.999187][ T69] #1: ffa00000004a7d10 (net_cleanup_work){+.+.}-{0:0}, at: process_one_work+0xdb7/0x1410 [ 485.999359][ T69] #2: ffffffffbd69e3a8 (pernet_ops_rwsem){++++}-{4:4}, at: cleanup_net+0xbf/0x890 [ 485.999568][ T69] #3: ffffffffbd6b9bc0 (rtnl_mutex){+.+.}-{4:4}, at: ops_undo_list+0x534/0x8f0 [ 485.999738][ T69] #4: ff1100001764d358 (&tbl->lock){+.-.}-{3:3}, at: __neigh_ifdown.isra.0+0xa1/0xea0 [ 485.999947][ T69] [ 485.999947][ T69] stack backtrace: [ 486.000060][ T69] CPU: 1 UID: 0 PID: 69 Comm: kworker/u16:1 Not tainted 7.2.0-rc6-virtme #1 PREEMPT(full) [ 486.000063][ T69] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 486.000065][ T69] Workqueue: netns cleanup_net [ 486.000068][ T69] Call Trace: [ 486.000070][ T69] [ 486.000072][ T69] dump_stack_lvl+0x6f/0xa0 [ 486.000075][ T69] print_circular_bug.cold+0x38/0x46 [ 486.000078][ T69] check_noncircular+0x171/0x1a0 [ 486.000081][ T69] check_prev_add+0xd8/0xe90 [ 486.000083][ T69] ? look_up_lock_class+0x65/0x150 [ 486.000085][ T69] ? check_prev_add+0xef/0xe90 [ 486.000088][ T69] validate_chain+0x6b8/0xc20 [ 486.000090][ T69] __lock_acquire+0x518/0xc20 [ 486.000093][ T69] lock_acquire.part.0+0xbc/0x260 [ 486.000095][ T69] ? __timer_delete_sync+0xdb/0x130 [ 486.000097][ T69] ? rcu_is_watching+0x15/0xd0 [ 486.000100][ T69] ? lock_acquire+0x13c/0x160 [ 486.000102][ T69] ? __timer_delete_sync+0xdb/0x130 [ 486.000103][ T69] __timer_delete_sync+0xea/0x130 [ 486.000104][ T69] ? __timer_delete_sync+0xdb/0x130 [ 486.000106][ T69] neigh_flush_one+0x4bf/0x650 [ 486.000108][ T69] __neigh_ifdown.isra.0+0x1b5/0xea0 [ 486.000110][ T69] ? fib6_clean_tree+0xf2/0x130 [ 486.000112][ T69] ? fib6_walk+0x3d0/0x3d0 [ 486.000114][ T69] ? neigh_flush_one+0x650/0x650 [ 486.000116][ T69] ? find_held_lock+0x2b/0x80 [ 486.000118][ T69] ? fib6_ifup+0x220/0x220 [ 486.000120][ T69] ? __lock_release.isra.0+0x6b/0x1a0 [ 486.000122][ T69] ? __fib6_clean_all+0x149/0x290 [ 486.000124][ T69] ? __local_bh_enable_ip+0xa5/0x140 [ 486.000126][ T69] ? __fib6_clean_all+0x14e/0x290 [ 486.000129][ T69] neigh_ifdown+0x10/0x20 [ 486.000130][ T69] rt6_disable_ip+0x183/0x200 [ 486.000132][ T69] ? rt6_sync_down_dev+0x120/0x120 [ 486.000134][ T69] ? __neigh_ifdown.isra.0+0x988/0xea0 [ 486.000136][ T69] addrconf_ifdown.isra.0+0x103/0x1670 [ 486.000139][ T69] ? neigh_flush_one+0x650/0x650 [ 486.000141][ T69] ? addrconf_sysctl_addr_gen_mode+0xb10/0xb10 [ 486.000143][ T69] ? mark_held_locks+0x40/0x70 [ 486.000146][ T69] addrconf_notify+0x2c8/0xf30 [ 486.000148][ T69] ? team_port_get_rtnl+0x65/0xc0 [ 486.000150][ T69] notifier_call_chain+0xae/0x300 [ 486.000153][ T69] netif_close_many+0x366/0x710 [ 486.000156][ T69] ? __dev_close_many+0x530/0x530 [ 486.000158][ T69] ? netif_close_many_and_unlock+0x22/0x2c0 [ 486.000160][ T69] unregister_netdevice_many_notify+0x671/0x19a0 [ 486.000162][ T69] ? mark_held_locks+0x40/0x70 [ 486.000164][ T69] ? unregister_netdevice_queued+0x80/0x80 [ 486.000165][ T69] ? lockdep_hardirqs_on+0x8c/0x130 [ 486.000167][ T69] ? __call_rcu_common.constprop.0+0x3d1/0xa10 [ 486.000169][ T69] ? mutex_is_locked+0x1c/0x50 [ 486.000171][ T69] ? rtnl_is_locked+0x15/0x20 [ 486.000173][ T69] ? nexthop_net_exit_rtnl+0x1f/0x270 [ 486.000177][ T69] ops_undo_list+0x615/0x8f0 [ 486.000179][ T69] ? lockdep_hardirqs_on+0x8c/0x130 [ 486.000180][ T69] ? rtnl_net_dumpid_one+0x2d0/0x2d0 [ 486.000182][ T69] ? __lock_release.isra.0+0x6b/0x1a0 [ 486.000184][ T69] ? idr_destroy+0x121/0x250 [ 486.000187][ T69] cleanup_net+0x431/0x890 [ 486.000189][ T69] ? net_passive_dec+0x1c0/0x1c0 [ 486.000190][ T69] ? process_one_work+0xdb7/0x1410 [ 486.000192][ T69] ? lock_acquire+0x13c/0x160 [ 486.000194][ T69] ? rcu_is_watching+0x15/0xd0 [ 486.000196][ T69] process_one_work+0xdf8/0x1410 [ 486.000198][ T69] ? pwq_dec_nr_in_flight+0x710/0x710 [ 486.000200][ T69] ? lock_acquire.part.0+0xbc/0x260 [ 486.000203][ T69] worker_thread+0x4f1/0xd60 [ 486.000205][ T69] ? rescuer_thread+0x1320/0x1320 [ 486.000206][ T69] ? __kthread_parkme+0xbd/0x210 [ 486.000208][ T69] ? rescuer_thread+0x1320/0x1320 [ 486.000210][ T69] ? rescuer_thread+0x1320/0x1320 [ 486.000211][ T69] kthread+0x367/0x460 [ 486.000212][ T69] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 486.000215][ T69] ? kthread_affine_node+0x330/0x330 [ 486.000217][ T69] ret_from_fork+0x474/0x6b0 [ 486.000219][ T69] ? arch_exit_to_user_mode_prepare.isra.0+0x120/0x120 [ 486.000221][ T69] ? __switch_to+0x5a3/0xe00 [ 486.000224][ T69] ? kthread_affine_node+0x330/0x330 [ 486.000225][ T69] ret_from_fork_asm+0x11/0x20 [ 486.000229][ T69] [ 486.088080][ T69] vx20: left allmulticast mode [ 486.088561][ T69] vx20: left promiscuous mode [ 486.090814][ T69] br2: port 3(vx20) entered disabled state [ 486.099238][ T69] vx10: left allmulticast mode [ 486.099392][ T69] vx10: left promiscuous mode [ 486.099597][ T69] br2: port 2(vx10) entered disabled state [ 486.100384][ T69] w1: left allmulticast mode [ 486.100519][ T69] w1: left promiscuous mode [ 486.100897][ T69] br2: port 1(w1) entered disabled state [ 486.589023][T10502] br1: port 4(veth2) entered disabled state [ 486.608217][T10503] veth2: left allmulticast mode [ 486.608418][T10503] veth2: left promiscuous mode [ 486.608727][T10503] br1: port 4(veth2) entered disabled state [ 486.683918][T10507] br1: port 3(veth1) entered disabled state [ 486.704058][T10508] veth1: left allmulticast mode [ 486.704212][T10508] veth1: left promiscuous mode [ 486.704421][T10508] br1: port 3(veth1) entered disabled state [ 486.741346][T10510] vx20: left allmulticast mode [ 486.741499][T10510] vx20: left promiscuous mode [ 486.741719][T10510] br1: port 2(vx20) entered disabled state [ 486.830474][T10514] vx10: left allmulticast mode [ 486.830654][T10514] vx10: left promiscuous mode [ 486.830892][T10514] br1: port 1(vx10) entered disabled state