[ 39.154036][ T1432] gre: GRE over IPv4 demultiplexer driver [ 39.170043][ T1432] ip_gre: GRE over IPv4 tunneling driver [ 45.214930][ T1468] ip6_gre: GRE over IPv6 tunneling driver [ 50.972419][ T67] [ 50.972500][ T67] ====================================================== [ 50.972625][ T67] WARNING: possible circular locking dependency detected [ 50.972735][ T67] 7.2.0-rc6-virtme #1 Not tainted [ 50.972828][ T67] ------------------------------------------------------ [ 50.972936][ T67] kworker/u16:1/67 is trying to acquire lock: [ 50.973040][ T67] ff11000010679d08 ((&n->timer)){+.-.}-{0:0}, at: __timer_delete_sync+0xdb/0x130 [ 50.973199][ T67] [ 50.973199][ T67] but task is already holding lock: [ 50.973323][ T67] ff1100000e710b58 (&tbl->lock){+.-.}-{3:3}, at: __neigh_ifdown.isra.0+0xa1/0xea0 [ 50.973474][ T67] [ 50.973474][ T67] which lock already depends on the new lock. [ 50.973474][ T67] [ 50.973639][ T67] [ 50.973639][ T67] the existing dependency chain (in reverse order) is: [ 50.973787][ T67] [ 50.973787][ T67] -> #1 (&tbl->lock){+.-.}-{3:3}: [ 50.973911][ T67] __lock_acquire+0x518/0xc20 [ 50.973995][ T67] lock_acquire.part.0+0xbc/0x260 [ 50.974079][ T67] _raw_spin_lock_bh+0x38/0x50 [ 50.974162][ T67] ___neigh_create+0x2be/0x1070 [ 50.974245][ T67] ip_finish_output2+0xf06/0x1f90 [ 50.974327][ T67] ip_output+0x1ca/0x660 [ 50.974409][ T67] ip_push_pending_frames+0xf6/0x250 [ 50.974510][ T67] __icmp_send+0xcb7/0x1ae0 [ 50.974596][ T67] ipv4_send_dest_unreach+0x438/0x9d0 [ 50.974696][ T67] ipv4_link_failure+0x16/0x190 [ 50.974782][ T67] arp_error_report+0x97/0x1b0 [ 50.974864][ T67] neigh_invalidate+0x208/0x560 [ 50.974943][ T67] neigh_timer_handler+0x8e9/0x1210 [ 50.975022][ T67] call_timer_fn+0x163/0x4c0 [ 50.975102][ T67] __run_timers+0x68f/0xaa0 [ 50.975182][ T67] run_timer_softirq+0xf0/0x160 [ 50.975261][ T67] handle_softirqs+0x1d8/0x8f0 [ 50.975345][ T67] __irq_exit_rcu+0x103/0x1c0 [ 50.975426][ T67] irq_exit_rcu+0xe/0x30 [ 50.975506][ T67] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 50.975607][ T67] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 50.975710][ T67] pv_native_safe_halt+0xf/0x10 [ 50.975795][ T67] default_idle+0x9/0x10 [ 50.975875][ T67] default_idle_call+0x6e/0xb0 [ 50.975956][ T67] cpuidle_idle_call.constprop.0+0x237/0x410 [ 50.976059][ T67] do_idle+0xd8/0x180 [ 50.976140][ T67] cpu_startup_entry+0x53/0x70 [ 50.976220][ T67] start_secondary+0x204/0x2b0 [ 50.976304][ T67] common_startup_64+0x13e/0x148 [ 50.976385][ T67] [ 50.976385][ T67] -> #0 ((&n->timer)){+.-.}-{0:0}: [ 50.976509][ T67] check_prev_add+0xd8/0xe90 [ 50.976595][ T67] validate_chain+0x6b8/0xc20 [ 50.976676][ T67] __lock_acquire+0x518/0xc20 [ 50.976761][ T67] lock_acquire.part.0+0xbc/0x260 [ 50.976841][ T67] __timer_delete_sync+0xea/0x130 [ 50.976920][ T67] neigh_flush_one+0x4bf/0x650 [ 50.976999][ T67] __neigh_ifdown.isra.0+0x1b5/0xea0 [ 50.977100][ T67] neigh_ifdown+0x10/0x20 [ 50.977181][ T67] rt6_disable_ip+0x183/0x200 [ 50.977262][ T67] addrconf_ifdown.isra.0+0x103/0x1670 [ 50.977363][ T67] addrconf_notify+0x2c8/0xf30 [ 50.977444][ T67] notifier_call_chain+0xae/0x300 [ 50.977525][ T67] netif_close_many+0x366/0x770 [ 50.977608][ T67] unregister_netdevice_many_notify+0x71a/0x1b40 [ 50.977710][ T67] ops_undo_list+0x615/0x8f0 [ 50.977793][ T67] cleanup_net+0x431/0x940 [ 50.977873][ T67] process_one_work+0xdf8/0x1410 [ 50.977956][ T67] worker_thread+0x4f1/0xd60 [ 50.978034][ T67] kthread+0x367/0x460 [ 50.978115][ T67] ret_from_fork+0x474/0x6b0 [ 50.978198][ T67] ret_from_fork_asm+0x11/0x20 [ 50.978283][ T67] [ 50.978283][ T67] other info that might help us debug this: [ 50.978283][ T67] [ 50.978445][ T67] Possible unsafe locking scenario: [ 50.978445][ T67] [ 50.978567][ T67] CPU0 CPU1 [ 50.978652][ T67] ---- ---- [ 50.978739][ T67] lock(&tbl->lock); [ 50.978809][ T67] lock((&n->timer)); [ 50.978914][ T67] lock(&tbl->lock); [ 50.979015][ T67] lock((&n->timer)); [ 50.979077][ T67] [ 50.979077][ T67] *** DEADLOCK *** [ 50.979077][ T67] [ 50.979195][ T67] 5 locks held by kworker/u16:1/67: [ 50.979275][ T67] #0: ff110000019f1d40 ((wq_completion)netns){+.+.}-{0:0}, at: process_one_work+0xd55/0x1410 [ 50.979439][ T67] #1: ffa0000000487d10 (net_cleanup_work){+.+.}-{0:0}, at: process_one_work+0xdb7/0x1410 [ 50.979583][ T67] #2: ffffffff8c6c5528 (pernet_ops_rwsem){++++}-{4:4}, at: cleanup_net+0xbf/0x940 [ 50.979723][ T67] #3: ffffffff8c6e0d40 (rtnl_mutex){+.+.}-{4:4}, at: ops_undo_list+0x534/0x8f0 [ 50.979869][ T67] #4: ff1100000e710b58 (&tbl->lock){+.-.}-{3:3}, at: __neigh_ifdown.isra.0+0xa1/0xea0 [ 50.980009][ T67] [ 50.980009][ T67] stack backtrace: [ 50.980109][ T67] CPU: 2 UID: 0 PID: 67 Comm: kworker/u16:1 Not tainted 7.2.0-rc6-virtme #1 PREEMPT(full) [ 50.980111][ T67] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 50.980114][ T67] Workqueue: netns cleanup_net [ 50.980116][ T67] Call Trace: [ 50.980118][ T67] [ 50.980120][ T67] dump_stack_lvl+0x6f/0xa0 [ 50.980123][ T67] print_circular_bug.cold+0x38/0x46 [ 50.980126][ T67] check_noncircular+0x171/0x1a0 [ 50.980130][ T67] check_prev_add+0xd8/0xe90 [ 50.980132][ T67] ? look_up_lock_class+0x65/0x150 [ 50.980134][ T67] ? check_prev_add+0xef/0xe90 [ 50.980136][ T67] validate_chain+0x6b8/0xc20 [ 50.980138][ T67] __lock_acquire+0x518/0xc20 [ 50.980141][ T67] lock_acquire.part.0+0xbc/0x260 [ 50.980143][ T67] ? __timer_delete_sync+0xdb/0x130 [ 50.980145][ T67] ? rcu_is_watching+0x15/0xd0 [ 50.980147][ T67] ? lock_acquire+0x13c/0x160 [ 50.980149][ T67] ? __timer_delete_sync+0xdb/0x130 [ 50.980150][ T67] __timer_delete_sync+0xea/0x130 [ 50.980152][ T67] ? __timer_delete_sync+0xdb/0x130 [ 50.980153][ T67] neigh_flush_one+0x4bf/0x650 [ 50.980155][ T67] __neigh_ifdown.isra.0+0x1b5/0xea0 [ 50.980157][ T67] ? fib6_clean_tree+0xf2/0x130 [ 50.980159][ T67] ? fib6_walk+0x3d0/0x3d0 [ 50.980161][ T67] ? neigh_flush_one+0x650/0x650 [ 50.980162][ T67] ? find_held_lock+0x2b/0x80 [ 50.980164][ T67] ? fib6_ifup+0x220/0x220 [ 50.980165][ T67] ? __lock_release.isra.0+0x6b/0x1a0 [ 50.980167][ T67] ? __fib6_clean_all+0x149/0x290 [ 50.980169][ T67] ? __local_bh_enable_ip+0xa5/0x140 [ 50.980171][ T67] ? __fib6_clean_all+0x14e/0x290 [ 50.980173][ T67] neigh_ifdown+0x10/0x20 [ 50.980175][ T67] rt6_disable_ip+0x183/0x200 [ 50.980177][ T67] ? rt6_sync_down_dev+0x120/0x120 [ 50.980178][ T67] ? __neigh_ifdown.isra.0+0x988/0xea0 [ 50.980180][ T67] addrconf_ifdown.isra.0+0x103/0x1670 [ 50.980182][ T67] ? neigh_flush_one+0x650/0x650 [ 50.980184][ T67] ? addrconf_sysctl_addr_gen_mode+0xd00/0xd00 [ 50.980186][ T67] ? netkit_xmit+0x14c0/0x14c0 [ 50.980190][ T67] ? mark_held_locks+0x40/0x70 [ 50.980192][ T67] addrconf_notify+0x2c8/0xf30 [ 50.980194][ T67] ? team_port_get_rtnl+0x65/0xc0 [ 50.980197][ T67] notifier_call_chain+0xae/0x300 [ 50.980200][ T67] netif_close_many+0x366/0x770 [ 50.980202][ T67] ? __dev_close_many+0x680/0x680 [ 50.980203][ T67] ? netif_close_many_and_unlock+0x22/0x2c0 [ 50.980205][ T67] unregister_netdevice_many_notify+0x71a/0x1b40 [ 50.980207][ T67] ? mark_held_locks+0x40/0x70 [ 50.980209][ T67] ? unregister_netdevice_queued+0x80/0x80 [ 50.980211][ T67] ? lockdep_hardirqs_on+0x8c/0x130 [ 50.980212][ T67] ? __call_rcu_common.constprop.0+0x3d1/0xa10 [ 50.980215][ T67] ? mutex_is_locked+0x1c/0x50 [ 50.980216][ T67] ? rtnl_is_locked+0x15/0x20 [ 50.980218][ T67] ? nexthop_net_exit_rtnl+0x1f/0x270 [ 50.980221][ T67] ? mutex_is_locked+0x1c/0x50 [ 50.980222][ T67] ops_undo_list+0x615/0x8f0 [ 50.980225][ T67] ? lockdep_hardirqs_on+0x8c/0x130 [ 50.980226][ T67] ? netns_install+0x2a0/0x2a0 [ 50.980228][ T67] ? __lock_release.isra.0+0x6b/0x1a0 [ 50.980230][ T67] ? idr_destroy+0x121/0x250 [ 50.980233][ T67] cleanup_net+0x431/0x940 [ 50.980234][ T67] ? net_passive_dec+0x1c0/0x1c0 [ 50.980236][ T67] ? process_one_work+0xdb7/0x1410 [ 50.980237][ T67] ? lock_acquire+0x13c/0x160 [ 50.980239][ T67] ? rcu_is_watching+0x15/0xd0 [ 50.980241][ T67] process_one_work+0xdf8/0x1410 [ 50.980243][ T67] ? pwq_dec_nr_in_flight+0x710/0x710 [ 50.980244][ T67] ? lock_acquire.part.0+0xbc/0x260 [ 50.980247][ T67] worker_thread+0x4f1/0xd60 [ 50.980249][ T67] ? rescuer_thread+0x1320/0x1320 [ 50.980250][ T67] ? __kthread_parkme+0xbd/0x210 [ 50.980253][ T67] ? rescuer_thread+0x1320/0x1320 [ 50.980254][ T67] ? rescuer_thread+0x1320/0x1320 [ 50.980255][ T67] kthread+0x367/0x460 [ 50.980256][ T67] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 50.980259][ T67] ? kthread_affine_node+0x330/0x330 [ 50.980260][ T67] ret_from_fork+0x474/0x6b0 [ 50.980262][ T67] ? arch_exit_to_user_mode_prepare.isra.0+0x120/0x120 [ 50.980264][ T67] ? __switch_to+0x5a3/0xe00 [ 50.980267][ T67] ? kthread_affine_node+0x330/0x330 [ 50.980268][ T67] ret_from_fork_asm+0x11/0x20 [ 50.980272][ T67]