[ 790.311180] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 790.314180] nipa-hw-worker: [5/47] Retrying drivers/net:napi_threaded.py
[ 790.449618] kselftest: Running tests in drivers/net
[ 989.672864] INFO: task python3:5891 blocked for more than 122 seconds.
[ 989.681082] Not tainted 7.1.0-rc3-fbrn-g82170046794c #1
[ 989.688334] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.698549] task:python3 state:D stack:25616 pid:5891 tgid:5891 ppid:5888 task_flags:0x400100 flags:0x00080000
[ 989.711758] Call Trace:
[ 989.715298]
[ 989.718451] __schedule+0xcc3/0x19c0
[ 989.723269] ? io_schedule_timeout+0x140/0x140
[ 989.729054] ? find_held_lock+0x2b/0x80
[ 989.734163] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.740046] schedule+0xd4/0x260
[ 989.744466] schedule_preempt_disabled+0x18/0x30
[ 989.750436] __mutex_lock+0x6a3/0x2000
[ 989.755442] ? threaded_store+0x132/0x220
[ 989.760743] ? ww_mutex_lock+0x160/0x160
[ 989.765942] ? selinux_inode_free_security+0x330/0x330
[ 989.772514] ? cap_capable+0x1d6/0x3e0
[ 989.777522] ? threaded_store+0x132/0x220
[ 989.782817] threaded_store+0x132/0x220
[ 989.787919] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.793219] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.798521] ? sysfs_kf_write+0x65/0x160
[ 989.803721] ? sysfs_file_ops+0x140/0x140
[ 989.809019] kernfs_fop_write_iter+0x335/0x510
[ 989.814806] ? kernfs_ops+0xf0/0xf0
[ 989.819517] new_sync_write+0x333/0x750
[ 989.824611] ? __lock_acquire+0x508/0xc10
[ 989.829911] ? new_sync_read+0x740/0x740
[ 989.835112] ? lock_acquire.part.0+0xbc/0x260
[ 989.840797] ? ksys_write+0x116/0x250
[ 989.845711] vfs_write+0x657/0x9b0
[ 989.850329] ksys_write+0x116/0x250
[ 989.855040] ? __ia32_sys_read+0xc0/0xc0
[ 989.860241] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.865832] ? rcu_is_watching+0x15/0xd0
[ 989.871036] do_syscall_64+0xf3/0x650
[ 989.875943] ? trace_hardirqs_off+0xd/0x30
[ 989.881339] ? exc_page_fault+0xda/0xf0
[ 989.886440] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.892908] RIP: 0033:0x7f05dd7cd22e
[ 989.897719] RSP: 002b:00007ffd59b66980 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.907017] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f05dd7cd22e
[ 989.915826] RDX: 0000000000000001 RSI: 0000563a85d37cd0 RDI: 0000000000000007
[ 989.924635] RBP: 00007ffd59b66990 R08: 0000000000000000 R09: 0000000000000000
[ 989.933445] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f05dd645390
[ 989.942252] R13: 0000563a85d37cd0 R14: 0000000000000007 R15: 0000000000000003
[ 989.951063]
[ 989.954380] INFO: task python3:5891 is blocked on a mutex likely owned by task python3:3835.
[ 989.964653] task:python3 state:D stack:25672 pid:3835 tgid:3835 ppid:3834 task_flags:0x400100 flags:0x00080802
[ 989.977855] Call Trace:
[ 989.981396]
[ 989.984546] __schedule+0xcc3/0x19c0
[ 989.989359] ? io_schedule_timeout+0x140/0x140
[ 989.995142] ? find_held_lock+0x2b/0x80
[ 990.000245] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.006426] schedule+0xd4/0x260
[ 990.010846] schedule_timeout+0x13f/0x290
[ 990.016145] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.022517] ? trace_contention_end+0xb3/0x1a0
[ 990.028300] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.033990] ? ww_mutex_lock+0x160/0x160
[ 990.039189] msleep+0x1b/0x30
[ 990.043305] napi_set_threaded+0x3a5/0x440
[ 990.048706] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.054202] netif_set_threaded+0x257/0x400
[ 990.059695] threaded_store+0x1b8/0x220
[ 990.064804] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.070101] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.075398] ? sysfs_kf_write+0x65/0x160
[ 990.080598] ? sysfs_file_ops+0x140/0x140
[ 990.085897] kernfs_fop_write_iter+0x335/0x510
[ 990.091683] ? kernfs_ops+0xf0/0xf0
[ 990.096396] new_sync_write+0x333/0x750
[ 990.101501] ? __lock_acquire+0x508/0xc10
[ 990.106798] ? new_sync_read+0x740/0x740
[ 990.111992] ? lock_acquire.part.0+0xbc/0x260
[ 990.117676] ? ksys_write+0x116/0x250
[ 990.122586] vfs_write+0x657/0x9b0
[ 990.127201] ksys_write+0x116/0x250
[ 990.131908] ? __ia32_sys_read+0xc0/0xc0
[ 990.137107] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.142696] ? rcu_is_watching+0x15/0xd0
[ 990.147902] do_syscall_64+0xf3/0x650
[ 990.152807] ? irq_exit_rcu+0x1a/0x30
[ 990.157718] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.164188] RIP: 0033:0x7f02a7e8422e
[ 990.168998] RSP: 002b:00007ffe81fb3550 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.178296] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f02a7e8422e
[ 990.187105] RDX: 0000000000000001 RSI: 0000555a8130bcd0 RDI: 0000000000000007
[ 990.195916] RBP: 00007ffe81fb3560 R08: 0000000000000000 R09: 0000000000000000
[ 990.204724] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f02a7cfc390
[ 990.213521] R13: 0000555a8130bcd0 R14: 0000000000000007 R15: 0000000000000003
[ 990.222337]
[ 990.225596] \x0aShowing all locks held in the system:
[ 990.234516] 1 lock held by khungtaskd/184:
[ 990.240132] #0: ffffffffa8efe060 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.251590] 2 locks held by kworker/4:1/213:
[ 990.257179] #0: ff110001000da740 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0xd52/0x1410
[ 990.268632] #1: ffa00000012cfd20 (key_gc_work){+.+.}-{0:0}, at: process_one_work+0xdb4/0x1410
[ 990.279138] 4 locks held by python3/3835:
[ 990.284432] #0: ff1100012f2a13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.294320] #1: ff11000188963280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.305281] #2: ff11000122382b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.316440] #3: ff11000122429d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.326541] 4 locks held by python3/5891:
[ 990.331836] #0: ff1100012f2a13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.341739] #1: ff110001f3568880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.352706] #2: ff11000122382b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.363862] #3: ff11000122429d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.373952]
[ 990.376429] =============================================\x0a
[ 1112.553063] INFO: task python3:5891 blocked for more than 245 seconds.
[ 1112.561204] Not tainted 7.1.0-rc3-fbrn-g82170046794c #1
[ 1112.568456] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.578044] task:python3 state:D stack:25616 pid:5891 tgid:5891 ppid:5888 task_flags:0x400100 flags:0x00080000
[ 1112.591249] Call Trace:
[ 1112.594791]
[ 1112.597940] __schedule+0xcc3/0x19c0
[ 1112.602755] ? io_schedule_timeout+0x140/0x140
[ 1112.608543] ? find_held_lock+0x2b/0x80
[ 1112.613646] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.619532] schedule+0xd4/0x260
[ 1112.623950] schedule_preempt_disabled+0x18/0x30
[ 1112.629928] __mutex_lock+0x6a3/0x2000
[ 1112.634931] ? threaded_store+0x132/0x220
[ 1112.640230] ? ww_mutex_lock+0x160/0x160
[ 1112.645431] ? selinux_inode_free_security+0x330/0x330
[ 1112.652002] ? cap_capable+0x1d6/0x3e0
[ 1112.657008] ? threaded_store+0x132/0x220
[ 1112.662308] threaded_store+0x132/0x220
[ 1112.667412] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.672711] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.678008] ? sysfs_kf_write+0x65/0x160
[ 1112.683210] ? sysfs_file_ops+0x140/0x140
[ 1112.688509] kernfs_fop_write_iter+0x335/0x510
[ 1112.694292] ? kernfs_ops+0xf0/0xf0
[ 1112.699005] new_sync_write+0x333/0x750
[ 1112.704111] ? __lock_acquire+0x508/0xc10
[ 1112.709408] ? new_sync_read+0x740/0x740
[ 1112.714611] ? lock_acquire.part.0+0xbc/0x260
[ 1112.720298] ? ksys_write+0x116/0x250
[ 1112.725208] vfs_write+0x657/0x9b0
[ 1112.729826] ksys_write+0x116/0x250
[ 1112.734540] ? __ia32_sys_read+0xc0/0xc0
[ 1112.739738] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.745325] ? rcu_is_watching+0x15/0xd0
[ 1112.750527] do_syscall_64+0xf3/0x650
[ 1112.755432] ? trace_hardirqs_off+0xd/0x30
[ 1112.760828] ? exc_page_fault+0xda/0xf0
[ 1112.765922] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.772389] RIP: 0033:0x7f05dd7cd22e
[ 1112.777200] RSP: 002b:00007ffd59b66980 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.786500] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f05dd7cd22e
[ 1112.795310] RDX: 0000000000000001 RSI: 0000563a85d37cd0 RDI: 0000000000000007
[ 1112.804120] RBP: 00007ffd59b66990 R08: 0000000000000000 R09: 0000000000000000
[ 1112.812927] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f05dd645390
[ 1112.821735] R13: 0000563a85d37cd0 R14: 0000000000000007 R15: 0000000000000003
[ 1112.830550]
[ 1112.833814] INFO: task python3:5891 is blocked on a mutex likely owned by task python3:3835.
[ 1112.844087] task:python3 state:D stack:25672 pid:3835 tgid:3835 ppid:3834 task_flags:0x400100 flags:0x00080802
[ 1112.857287] Call Trace:
[ 1112.860826]
[ 1112.863979] __schedule+0xcc3/0x19c0
[ 1112.868792] ? io_schedule_timeout+0x140/0x140
[ 1112.874577] ? find_held_lock+0x2b/0x80
[ 1112.879683] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.885862] schedule+0xd4/0x260
[ 1112.890282] schedule_timeout+0x13f/0x290
[ 1112.895580] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.901951] ? trace_contention_end+0xb3/0x1a0
[ 1112.907735] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.913427] ? ww_mutex_lock+0x160/0x160
[ 1112.918625] msleep+0x1b/0x30
[ 1112.922751] napi_set_threaded+0x3a5/0x440
[ 1112.928138] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.933633] netif_set_threaded+0x257/0x400
[ 1112.939129] threaded_store+0x1b8/0x220
[ 1112.944231] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.949528] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.954824] ? sysfs_kf_write+0x65/0x160
[ 1112.960025] ? sysfs_file_ops+0x140/0x140
[ 1112.965323] kernfs_fop_write_iter+0x335/0x510
[ 1112.971107] ? kernfs_ops+0xf0/0xf0
[ 1112.975819] new_sync_write+0x333/0x750
[ 1112.980914] ? __lock_acquire+0x508/0xc10
[ 1112.986212] ? new_sync_read+0x740/0x740
[ 1112.991415] ? lock_acquire.part.0+0xbc/0x260
[ 1112.997102] ? ksys_write+0x116/0x250
[ 1113.002004] vfs_write+0x657/0x9b0
[ 1113.006619] ksys_write+0x116/0x250
[ 1113.011325] ? __ia32_sys_read+0xc0/0xc0
[ 1113.016524] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.022115] ? rcu_is_watching+0x15/0xd0
[ 1113.027315] do_syscall_64+0xf3/0x650
[ 1113.032222] ? irq_exit_rcu+0x1a/0x30
[ 1113.037131] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.043592] RIP: 0033:0x7f02a7e8422e
[ 1113.048403] RSP: 002b:00007ffe81fb3550 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.057702] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f02a7e8422e
[ 1113.066524] RDX: 0000000000000001 RSI: 0000555a8130bcd0 RDI: 0000000000000007
[ 1113.075350] RBP: 00007ffe81fb3560 R08: 0000000000000000 R09: 0000000000000000
[ 1113.084162] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f02a7cfc390
[ 1113.092972] R13: 0000555a8130bcd0 R14: 0000000000000007 R15: 0000000000000003
[ 1113.101786]
[ 1113.105052] \x0aShowing all locks held in the system:
[ 1113.113610] 1 lock held by khungtaskd/184:
[ 1113.119006] #0: ffffffffa8efe060 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.130480] 2 locks held by systemd-journal/805:
[ 1113.136458] #0: ff1100180e2372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1113.146345] #1: ffffffffa91033a0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1113.159164] 4 locks held by python3/3835:
[ 1113.164458] #0: ff1100012f2a13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.174347] #1: ff11000188963280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.185310] #2: ff11000122382b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.196467] #3: ff11000122429d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.206559] 4 locks held by python3/5891:
[ 1113.211854] #0: ff1100012f2a13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.221745] #1: ff110001f3568880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.232707] #2: ff11000122382b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.243864] #3: ff11000122429d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.253955]
[ 1113.256937] =============================================\x0a
[ 1235.435228] INFO: task python3:5891 blocked for more than 368 seconds.
[ 1235.443366] Not tainted 7.1.0-rc3-fbrn-g82170046794c #1
[ 1235.450617] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.460205] task:python3 state:D stack:25616 pid:5891 tgid:5891 ppid:5888 task_flags:0x400100 flags:0x00080002
[ 1235.473410] Call Trace:
[ 1235.476949]
[ 1235.480097] __schedule+0xcc3/0x19c0
[ 1235.484911] ? io_schedule_timeout+0x140/0x140
[ 1235.490694] ? find_held_lock+0x2b/0x80
[ 1235.495796] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.501682] schedule+0xd4/0x260
[ 1235.506098] schedule_preempt_disabled+0x18/0x30
[ 1235.512075] __mutex_lock+0x6a3/0x2000
[ 1235.517078] ? threaded_store+0x132/0x220
[ 1235.522368] ? ww_mutex_lock+0x160/0x160
[ 1235.527568] ? selinux_inode_free_security+0x330/0x330
[ 1235.534135] ? cap_capable+0x1d6/0x3e0
[ 1235.539140] ? threaded_store+0x132/0x220
[ 1235.544435] threaded_store+0x132/0x220
[ 1235.549537] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.554833] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.560133] ? sysfs_kf_write+0x65/0x160
[ 1235.565331] ? sysfs_file_ops+0x140/0x140
[ 1235.570625] kernfs_fop_write_iter+0x335/0x510
[ 1235.576408] ? kernfs_ops+0xf0/0xf0
[ 1235.581115] new_sync_write+0x333/0x750
[ 1235.586218] ? __lock_acquire+0x508/0xc10
[ 1235.591514] ? new_sync_read+0x740/0x740
[ 1235.596705] ? lock_acquire.part.0+0xbc/0x260
[ 1235.602388] ? ksys_write+0x116/0x250
[ 1235.607299] vfs_write+0x657/0x9b0
[ 1235.611913] ksys_write+0x116/0x250
[ 1235.616623] ? __ia32_sys_read+0xc0/0xc0
[ 1235.621822] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.627410] ? rcu_is_watching+0x15/0xd0
[ 1235.632613] do_syscall_64+0xf3/0x650
[ 1235.637518] ? trace_hardirqs_off+0xd/0x30
[ 1235.642912] ? exc_page_fault+0xda/0xf0
[ 1235.648012] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.654479] RIP: 0033:0x7f05dd7cd22e
[ 1235.659287] RSP: 002b:00007ffd59b66980 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.668583] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f05dd7cd22e
[ 1235.677391] RDX: 0000000000000001 RSI: 0000563a85d37cd0 RDI: 0000000000000007
[ 1235.686190] RBP: 00007ffd59b66990 R08: 0000000000000000 R09: 0000000000000000
[ 1235.694998] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f05dd645390
[ 1235.703806] R13: 0000563a85d37cd0 R14: 0000000000000007 R15: 0000000000000003
[ 1235.712609]
[ 1235.715900] INFO: task python3:5891 is blocked on a mutex likely owned by task python3:3835.
[ 1235.726173] task:python3 state:D stack:25672 pid:3835 tgid:3835 ppid:3834 task_flags:0x400100 flags:0x00080802
[ 1235.739375] Call Trace:
[ 1235.742910]
[ 1235.746062] __schedule+0xcc3/0x19c0
[ 1235.750863] ? io_schedule_timeout+0x140/0x140
[ 1235.756637] ? find_held_lock+0x2b/0x80
[ 1235.761738] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.767909] schedule+0xd4/0x260
[ 1235.772325] schedule_timeout+0x13f/0x290
[ 1235.777622] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.783989] ? trace_contention_end+0xb3/0x1a0
[ 1235.789774] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.795456] ? ww_mutex_lock+0x160/0x160
[ 1235.800654] msleep+0x1b/0x30
[ 1235.804778] napi_set_threaded+0x3a5/0x440
[ 1235.810174] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.815661] netif_set_threaded+0x257/0x400
[ 1235.821154] threaded_store+0x1b8/0x220
[ 1235.826252] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.831550] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.836843] ? sysfs_kf_write+0x65/0x160
[ 1235.842042] ? sysfs_file_ops+0x140/0x140
[ 1235.847337] kernfs_fop_write_iter+0x335/0x510
[ 1235.853119] ? kernfs_ops+0xf0/0xf0
[ 1235.857828] new_sync_write+0x333/0x750
[ 1235.862929] ? __lock_acquire+0x508/0xc10
[ 1235.868222] ? new_sync_read+0x740/0x740
[ 1235.873423] ? lock_acquire.part.0+0xbc/0x260
[ 1235.879108] ? ksys_write+0x116/0x250
[ 1235.884016] vfs_write+0x657/0x9b0
[ 1235.888633] ksys_write+0x116/0x250
[ 1235.893344] ? __ia32_sys_read+0xc0/0xc0
[ 1235.898541] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.904130] ? rcu_is_watching+0x15/0xd0
[ 1235.909328] do_syscall_64+0xf3/0x650
[ 1235.914232] ? irq_exit_rcu+0x1a/0x30
[ 1235.919138] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.925602] RIP: 0033:0x7f02a7e8422e
[ 1235.930409] RSP: 002b:00007ffe81fb3550 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.939705] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f02a7e8422e
[ 1235.948513] RDX: 0000000000000001 RSI: 0000555a8130bcd0 RDI: 0000000000000007
[ 1235.957312] RBP: 00007ffe81fb3560 R08: 0000000000000000 R09: 0000000000000000
[ 1235.966118] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f02a7cfc390
[ 1235.974924] R13: 0000555a8130bcd0 R14: 0000000000000007 R15: 0000000000000003
[ 1235.983735]
[ 1235.986996] \x0aShowing all locks held in the system:
[ 1235.995536] 1 lock held by khungtaskd/184:
[ 1236.000928] #0: ffffffffa8efe060 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.012410] 4 locks held by python3/3835:
[ 1236.017707] #0: ff1100012f2a13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.027596] #1: ff11000188963280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.038558] #2: ff11000122382b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.049714] #3: ff11000122429d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.059810] 4 locks held by python3/5891:
[ 1236.065105] #0: ff1100012f2a13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.074991] #1: ff110001f3568880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.085951] #2: ff11000122382b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.097106] #3: ff11000122429d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.107196]
[ 1236.109655] =============================================\x0a
[ 1288.683705] kworker/u129:0 (177) used greatest stack depth: 21936 bytes left
[ 1358.316402] INFO: task python3:5891 blocked for more than 491 seconds.
[ 1358.324548] Not tainted 7.1.0-rc3-fbrn-g82170046794c #1
[ 1358.331796] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.341384] task:python3 state:D stack:25616 pid:5891 tgid:5891 ppid:5888 task_flags:0x400100 flags:0x00080002
[ 1358.354588] Call Trace:
[ 1358.358124]
[ 1358.361273] __schedule+0xcc3/0x19c0
[ 1358.366087] ? io_schedule_timeout+0x140/0x140
[ 1358.371864] ? find_held_lock+0x2b/0x80
[ 1358.376967] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.382851] schedule+0xd4/0x260
[ 1358.387267] schedule_preempt_disabled+0x18/0x30
[ 1358.393244] __mutex_lock+0x6a3/0x2000
[ 1358.398247] ? threaded_store+0x132/0x220
[ 1358.403536] ? ww_mutex_lock+0x160/0x160
[ 1358.408732] ? selinux_inode_free_security+0x330/0x330
[ 1358.415302] ? cap_capable+0x1d6/0x3e0
[ 1358.420307] ? threaded_store+0x132/0x220
[ 1358.425601] threaded_store+0x132/0x220
[ 1358.430703] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.435997] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.441298] ? sysfs_kf_write+0x65/0x160
[ 1358.446497] ? sysfs_file_ops+0x140/0x140
[ 1358.451790] kernfs_fop_write_iter+0x335/0x510
[ 1358.457573] ? kernfs_ops+0xf0/0xf0
[ 1358.462280] new_sync_write+0x333/0x750
[ 1358.467382] ? __lock_acquire+0x508/0xc10
[ 1358.472679] ? new_sync_read+0x740/0x740
[ 1358.477877] ? lock_acquire.part.0+0xbc/0x260
[ 1358.483564] ? ksys_write+0x116/0x250
[ 1358.488465] vfs_write+0x657/0x9b0
[ 1358.493079] ksys_write+0x116/0x250
[ 1358.497789] ? __ia32_sys_read+0xc0/0xc0
[ 1358.502987] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.508574] ? rcu_is_watching+0x15/0xd0
[ 1358.513776] do_syscall_64+0xf3/0x650
[ 1358.518680] ? trace_hardirqs_off+0xd/0x30
[ 1358.524073] ? exc_page_fault+0xda/0xf0
[ 1358.529176] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.535642] RIP: 0033:0x7f05dd7cd22e
[ 1358.540449] RSP: 002b:00007ffd59b66980 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.549744] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f05dd7cd22e
[ 1358.558552] RDX: 0000000000000001 RSI: 0000563a85d37cd0 RDI: 0000000000000007
[ 1358.567350] RBP: 00007ffd59b66990 R08: 0000000000000000 R09: 0000000000000000
[ 1358.576158] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f05dd645390
[ 1358.584956] R13: 0000563a85d37cd0 R14: 0000000000000007 R15: 0000000000000003
[ 1358.593767]
[ 1358.597045] INFO: task python3:5891 is blocked on a mutex likely owned by task python3:3835.
[ 1358.607318] task:python3 state:D stack:25672 pid:3835 tgid:3835 ppid:3834 task_flags:0x400100 flags:0x00080802
[ 1358.620521] Call Trace:
[ 1358.624056]
[ 1358.627205] __schedule+0xcc3/0x19c0
[ 1358.632015] ? io_schedule_timeout+0x140/0x140
[ 1358.637799] ? find_held_lock+0x2b/0x80
[ 1358.642902] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.649077] schedule+0xd4/0x260
[ 1358.653494] schedule_timeout+0x13f/0x290
[ 1358.658790] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.665157] ? trace_contention_end+0xb3/0x1a0
[ 1358.670942] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.676630] ? ww_mutex_lock+0x160/0x160
[ 1358.681828] msleep+0x1b/0x30
[ 1358.685951] napi_set_threaded+0x3a5/0x440
[ 1358.691347] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.696831] netif_set_threaded+0x257/0x400
[ 1358.702324] threaded_store+0x1b8/0x220
[ 1358.707424] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.712720] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.718014] ? sysfs_kf_write+0x65/0x160
[ 1358.723212] ? sysfs_file_ops+0x140/0x140
[ 1358.728508] kernfs_fop_write_iter+0x335/0x510
[ 1358.734294] ? kernfs_ops+0xf0/0xf0
[ 1358.738995] new_sync_write+0x333/0x750
[ 1358.744095] ? __lock_acquire+0x508/0xc10
[ 1358.749388] ? new_sync_read+0x740/0x740
[ 1358.754588] ? lock_acquire.part.0+0xbc/0x260
[ 1358.760274] ? ksys_write+0x116/0x250
[ 1358.765183] vfs_write+0x657/0x9b0
[ 1358.769799] ksys_write+0x116/0x250
[ 1358.774502] ? __ia32_sys_read+0xc0/0xc0
[ 1358.779689] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.785275] ? rcu_is_watching+0x15/0xd0
[ 1358.790474] do_syscall_64+0xf3/0x650
[ 1358.795378] ? irq_exit_rcu+0x1a/0x30
[ 1358.800285] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.806749] RIP: 0033:0x7f02a7e8422e
[ 1358.811557] RSP: 002b:00007ffe81fb3550 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.820852] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f02a7e8422e
[ 1358.829656] RDX: 0000000000000001 RSI: 0000555a8130bcd0 RDI: 0000000000000007
[ 1358.838465] RBP: 00007ffe81fb3560 R08: 0000000000000000 R09: 0000000000000000
[ 1358.847269] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f02a7cfc390
[ 1358.856077] R13: 0000555a8130bcd0 R14: 0000000000000007 R15: 0000000000000003
[ 1358.864886]
[ 1358.868150] \x0aShowing all locks held in the system:
[ 1358.876688] 1 lock held by khungtaskd/184:
[ 1358.882078] #0: ffffffffa8efe060 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.893558] 4 locks held by python3/3835:
[ 1358.898850] #0: ff1100012f2a13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.908740] #1: ff11000188963280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.919700] #2: ff11000122382b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.930857] #3: ff11000122429d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.940939] 4 locks held by python3/5891:
[ 1358.946225] #0: ff1100012f2a13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.956117] #1: ff110001f3568880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.967077] #2: ff11000122382b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.978238] #3: ff11000122429d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.988328]
[ 1358.990783] =============================================\x0a