[ 9.999385][ T228] veth0: renamed from veth-rt-1 [ 10.310582][ T237] veth0: renamed from veth-rt-2 [ 10.871670][ T250] ip (250) used greatest stack depth: 23688 bytes left [ 41.085426][ T69] [ 41.085513][ T69] ====================================================== [ 41.085618][ T69] WARNING: possible circular locking dependency detected [ 41.085721][ T69] 7.2.0-rc6-virtme #1 Not tainted [ 41.085806][ T69] ------------------------------------------------------ [ 41.085906][ T69] kworker/u16:1/69 is trying to acquire lock: [ 41.086005][ T69] ff11000012727108 ((&n->timer)){+.-.}-{0:0}, at: __timer_delete_sync+0xdb/0x130 [ 41.086171][ T69] [ 41.086171][ T69] but task is already holding lock: [ 41.086291][ T69] ff1100000f6c6b58 (&tbl->lock){+.-.}-{3:3}, at: __neigh_ifdown.isra.0+0xa1/0xea0 [ 41.086439][ T69] [ 41.086439][ T69] which lock already depends on the new lock. [ 41.086439][ T69] [ 41.086600][ T69] [ 41.086600][ T69] the existing dependency chain (in reverse order) is: [ 41.086744][ T69] [ 41.086744][ T69] -> #1 (&tbl->lock){+.-.}-{3:3}: [ 41.086863][ T69] __lock_acquire+0x518/0xc20 [ 41.086946][ T69] lock_acquire.part.0+0xbc/0x260 [ 41.087030][ T69] _raw_spin_lock_bh+0x38/0x50 [ 41.087108][ T69] ___neigh_create+0x2be/0x1070 [ 41.087190][ T69] ip_finish_output2+0xf06/0x1f90 [ 41.087271][ T69] ip_output+0x1ca/0x660 [ 41.087353][ T69] ip_push_pending_frames+0xf6/0x250 [ 41.087452][ T69] __icmp_send+0xcb7/0x1ae0 [ 41.087534][ T69] ipv4_send_dest_unreach+0x438/0x9d0 [ 41.087635][ T69] ipv4_link_failure+0x16/0x190 [ 41.087717][ T69] arp_error_report+0x97/0x1b0 [ 41.087797][ T69] neigh_invalidate+0x208/0x560 [ 41.087875][ T69] neigh_timer_handler+0x8e9/0x1210 [ 41.087957][ T69] call_timer_fn+0x163/0x4c0 [ 41.088037][ T69] __run_timers+0x68f/0xaa0 [ 41.088114][ T69] run_timer_softirq+0xf0/0x160 [ 41.088195][ T69] handle_softirqs+0x1d8/0x8f0 [ 41.088276][ T69] __irq_exit_rcu+0x103/0x1c0 [ 41.088360][ T69] irq_exit_rcu+0xe/0x30 [ 41.088438][ T69] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 41.088542][ T69] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 41.088643][ T69] pv_native_safe_halt+0xf/0x10 [ 41.088725][ T69] default_idle+0x9/0x10 [ 41.088806][ T69] default_idle_call+0x6e/0xb0 [ 41.088885][ T69] cpuidle_idle_call.constprop.0+0x237/0x410 [ 41.089004][ T69] do_idle+0xd8/0x180 [ 41.089086][ T69] cpu_startup_entry+0x53/0x70 [ 41.089162][ T69] rest_init+0x1f7/0x200 [ 41.089239][ T69] start_kernel+0x3af/0x3b0 [ 41.089322][ T69] x86_64_start_reservations+0x24/0x30 [ 41.089416][ T69] x86_64_start_kernel+0x12b/0x130 [ 41.089508][ T69] common_startup_64+0x13e/0x148 [ 41.089588][ T69] [ 41.089588][ T69] -> #0 ((&n->timer)){+.-.}-{0:0}: [ 41.089707][ T69] check_prev_add+0xd8/0xe90 [ 41.089786][ T69] validate_chain+0x6b8/0xc20 [ 41.089869][ T69] __lock_acquire+0x518/0xc20 [ 41.089951][ T69] lock_acquire.part.0+0xbc/0x260 [ 41.090034][ T69] __timer_delete_sync+0xea/0x130 [ 41.090121][ T69] neigh_flush_one+0x4bf/0x650 [ 41.090198][ T69] __neigh_ifdown.isra.0+0x1b5/0xea0 [ 41.090303][ T69] neigh_ifdown+0x10/0x20 [ 41.090385][ T69] fib_netdev_event+0x1a9/0x5d0 [ 41.090463][ T69] notifier_call_chain+0xae/0x300 [ 41.090551][ T69] __netdev_upper_dev_unlink+0x10b/0x1e0 [ 41.090648][ T69] netdev_upper_dev_unlink+0x79/0xa0 [ 41.090744][ T69] vrf_dellink+0xcb/0x180 [ 41.090866][ T69] default_device_exit_batch+0x19d/0x600 [ 41.090962][ T69] ops_undo_list+0x2ce/0x8f0 [ 41.091040][ T69] cleanup_net+0x431/0x940 [ 41.091159][ T69] process_one_work+0xdf8/0x1410 [ 41.091242][ T69] worker_thread+0x4f1/0xd60 [ 41.091327][ T69] kthread+0x367/0x460 [ 41.091403][ T69] ret_from_fork+0x474/0x6b0 [ 41.091525][ T69] ret_from_fork_asm+0x11/0x20 [ 41.091610][ T69] [ 41.091610][ T69] other info that might help us debug this: [ 41.091610][ T69] [ 41.091767][ T69] Possible unsafe locking scenario: [ 41.091767][ T69] [ 41.091934][ T69] CPU0 CPU1 [ 41.092015][ T69] ---- ---- [ 41.092096][ T69] lock(&tbl->lock); [ 41.092157][ T69] lock((&n->timer)); [ 41.092301][ T69] lock(&tbl->lock); [ 41.092403][ T69] lock((&n->timer)); [ 41.092468][ T69] [ 41.092468][ T69] *** DEADLOCK *** [ 41.092468][ T69] [ 41.092627][ T69] 5 locks held by kworker/u16:1/69: [ 41.092710][ T69] #0: ff110000019f1d40 ((wq_completion)netns){+.+.}-{0:0}, at: process_one_work+0xd55/0x1410 [ 41.092867][ T69] #1: ffa00000004a7d10 (net_cleanup_work){+.+.}-{0:0}, at: process_one_work+0xdb7/0x1410 [ 41.093045][ T69] #2: ffffffff922c5528 (pernet_ops_rwsem){++++}-{4:4}, at: cleanup_net+0xbf/0x940 [ 41.093181][ T69] #3: ffffffff922e0d40 (rtnl_mutex){+.+.}-{4:4}, at: default_device_exit_batch+0x9a/0x600 [ 41.093385][ T69] #4: ff1100000f6c6b58 (&tbl->lock){+.-.}-{3:3}, at: __neigh_ifdown.isra.0+0xa1/0xea0 [ 41.093527][ T69] [ 41.093527][ T69] stack backtrace: [ 41.093625][ T69] CPU: 3 UID: 0 PID: 69 Comm: kworker/u16:1 Not tainted 7.2.0-rc6-virtme #1 PREEMPT(full) [ 41.093628][ T69] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 41.093630][ T69] Workqueue: netns cleanup_net [ 41.093633][ T69] Call Trace: [ 41.093634][ T69] [ 41.093636][ T69] dump_stack_lvl+0x6f/0xa0 [ 41.093639][ T69] print_circular_bug.cold+0x38/0x46 [ 41.093642][ T69] check_noncircular+0x171/0x1a0 [ 41.093645][ T69] check_prev_add+0xd8/0xe90 [ 41.093647][ T69] ? look_up_lock_class+0x65/0x150 [ 41.093649][ T69] ? check_prev_add+0x316/0xe90 [ 41.093651][ T69] validate_chain+0x6b8/0xc20 [ 41.093654][ T69] __lock_acquire+0x518/0xc20 [ 41.093656][ T69] lock_acquire.part.0+0xbc/0x260 [ 41.093658][ T69] ? __timer_delete_sync+0xdb/0x130 [ 41.093660][ T69] ? rcu_is_watching+0x15/0xd0 [ 41.093662][ T69] ? lock_acquire+0x13c/0x160 [ 41.093664][ T69] ? __timer_delete_sync+0xdb/0x130 [ 41.093666][ T69] __timer_delete_sync+0xea/0x130 [ 41.093667][ T69] ? __timer_delete_sync+0xdb/0x130 [ 41.093668][ T69] neigh_flush_one+0x4bf/0x650 [ 41.093670][ T69] __neigh_ifdown.isra.0+0x1b5/0xea0 [ 41.093672][ T69] ? __wake_up+0x44/0x60 [ 41.093674][ T69] ? netlink_broadcast_filtered+0x2cb/0x3c0 [ 41.093677][ T69] ? neigh_flush_one+0x650/0x650 [ 41.093678][ T69] ? do_one_broadcast+0x1130/0x1130 [ 41.093680][ T69] ? fib_sync_mtu+0x110/0x110 [ 41.093682][ T69] ? rtmsg_ifinfo_build_skb+0x144/0x250 [ 41.093685][ T69] neigh_ifdown+0x10/0x20 [ 41.093686][ T69] fib_netdev_event+0x1a9/0x5d0 [ 41.093689][ T69] notifier_call_chain+0xae/0x300 [ 41.093691][ T69] __netdev_upper_dev_unlink+0x10b/0x1e0 [ 41.093693][ T69] ? lock_acquire+0x13c/0x160 [ 41.093694][ T69] ? netdev_adjacent_change_prepare+0x170/0x170 [ 41.093696][ T69] ? trace_contention_end+0xb3/0x180 [ 41.093698][ T69] ? default_device_exit_batch+0x9a/0x600 [ 41.093700][ T69] ? rcu_is_watching+0x15/0xd0 [ 41.093701][ T69] ? lock_acquire+0x13c/0x160 [ 41.093703][ T69] ? rcu_is_watching+0x15/0xd0 [ 41.093704][ T69] netdev_upper_dev_unlink+0x79/0xa0 [ 41.093706][ T69] ? __netdev_upper_dev_unlink+0x1e0/0x1e0 [ 41.093708][ T69] vrf_dellink+0xcb/0x180 [ 41.093709][ T69] ? vrf_map_unregister_dev+0x410/0x410 [ 41.093711][ T69] ? perf_trace_sched_switch+0x7c0/0x7c0 [ 41.093714][ T69] default_device_exit_batch+0x19d/0x600 [ 41.093716][ T69] ? unregister_netdev+0x60/0x60 [ 41.093718][ T69] ? perf_trace_sched_switch+0x7c0/0x7c0 [ 41.093719][ T69] ? fou_exit_net+0x30/0x110 [ 41.093722][ T69] ops_undo_list+0x2ce/0x8f0 [ 41.093724][ T69] ? lockdep_hardirqs_on+0x8c/0x130 [ 41.093725][ T69] ? netns_install+0x2a0/0x2a0 [ 41.093727][ T69] ? __lock_release.isra.0+0x6b/0x1a0 [ 41.093729][ T69] ? idr_destroy+0x121/0x250 [ 41.093732][ T69] cleanup_net+0x431/0x940 [ 41.093733][ T69] ? net_passive_dec+0x1c0/0x1c0 [ 41.093735][ T69] ? process_one_work+0xdb7/0x1410 [ 41.093736][ T69] ? lock_acquire+0x13c/0x160 [ 41.093738][ T69] ? rcu_is_watching+0x15/0xd0 [ 41.093740][ T69] process_one_work+0xdf8/0x1410 [ 41.093742][ T69] ? pwq_dec_nr_in_flight+0x710/0x710 [ 41.093743][ T69] ? lock_acquire.part.0+0xbc/0x260 [ 41.093746][ T69] worker_thread+0x4f1/0xd60 [ 41.093748][ T69] ? rescuer_thread+0x1320/0x1320 [ 41.093749][ T69] ? __kthread_parkme+0xbd/0x210 [ 41.093752][ T69] ? rescuer_thread+0x1320/0x1320 [ 41.093753][ T69] ? rescuer_thread+0x1320/0x1320 [ 41.093754][ T69] kthread+0x367/0x460 [ 41.093755][ T69] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 41.093758][ T69] ? kthread_affine_node+0x330/0x330 [ 41.093759][ T69] ret_from_fork+0x474/0x6b0 [ 41.093761][ T69] ? arch_exit_to_user_mode_prepare.isra.0+0x120/0x120 [ 41.093763][ T69] ? __switch_to+0x5a3/0xe00 [ 41.093765][ T69] ? kthread_affine_node+0x330/0x330 [ 41.093766][ T69] ret_from_fork_asm+0x11/0x20 [ 41.093770][ T69]