[ 788.222208] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 788.225541] nipa-hw-worker: [5/46] Retrying drivers/net:napi_threaded.py
[ 788.406720] kselftest: Running tests in drivers/net
[ 989.716503] INFO: task python3:5815 blocked for more than 122 seconds.
[ 989.724745] Not tainted 7.1.0-rc4-zuul-g5f6251e63d83 #1
[ 989.732676] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.742268] task:python3 state:D stack:25672 pid:5815 tgid:5815 ppid:5814 task_flags:0x400100 flags:0x00080000
[ 989.755473] Call Trace:
[ 989.759015]
[ 989.762170] __schedule+0xcc3/0x19c0
[ 989.766989] ? io_schedule_timeout+0x140/0x140
[ 989.772776] ? find_held_lock+0x2b/0x80
[ 989.777883] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.783768] schedule+0xd8/0x260
[ 989.788185] schedule_preempt_disabled+0x18/0x30
[ 989.794164] __mutex_lock+0x6af/0x2010
[ 989.799170] ? threaded_store+0x132/0x220
[ 989.804470] ? ww_mutex_lock+0x160/0x160
[ 989.809661] ? selinux_inode_free_security+0x330/0x330
[ 989.816231] ? cap_capable+0x1d6/0x3e0
[ 989.821243] ? threaded_store+0x132/0x220
[ 989.826542] threaded_store+0x132/0x220
[ 989.831645] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.836935] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.842234] ? sysfs_kf_write+0x65/0x160
[ 989.847435] ? sysfs_file_ops+0x140/0x140
[ 989.852729] kernfs_fop_write_iter+0x335/0x510
[ 989.858513] ? kernfs_ops+0xf0/0xf0
[ 989.863226] new_sync_write+0x333/0x750
[ 989.868333] ? __lock_acquire+0x508/0xc10
[ 989.873631] ? new_sync_read+0x740/0x740
[ 989.878831] ? lock_acquire.part.0+0xbc/0x260
[ 989.884510] ? ksys_write+0x116/0x250
[ 989.889412] vfs_write+0x657/0x9b0
[ 989.894027] ksys_write+0x116/0x250
[ 989.898733] ? __ia32_sys_read+0xc0/0xc0
[ 989.903931] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.909521] ? rcu_is_watching+0x15/0xd0
[ 989.914723] do_syscall_64+0xf3/0x650
[ 989.919629] ? trace_hardirqs_off+0xd/0x30
[ 989.925027] ? exc_page_fault+0xda/0xf0
[ 989.930130] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.936600] RIP: 0033:0x7f267645d22e
[ 989.941409] RSP: 002b:00007fffe6448440 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.950706] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f267645d22e
[ 989.959505] RDX: 0000000000000001 RSI: 0000558cd3c6a6b0 RDI: 0000000000000007
[ 989.968304] RBP: 00007fffe6448450 R08: 0000000000000000 R09: 0000000000000000
[ 989.977105] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f26762d5390
[ 989.985910] R13: 0000558cd3c6a6b0 R14: 0000000000000007 R15: 0000000000000003
[ 989.994731]
[ 989.998017] INFO: task python3:5815 is blocked on a mutex likely owned by task python3:3752.
[ 990.008291] task:python3 state:D stack:25616 pid:3752 tgid:3752 ppid:3751 task_flags:0x400100 flags:0x00080802
[ 990.021491] Call Trace:
[ 990.025032]
[ 990.028181] __schedule+0xcc3/0x19c0
[ 990.032994] ? io_schedule_timeout+0x140/0x140
[ 990.038780] ? find_held_lock+0x2b/0x80
[ 990.043884] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.050062] schedule+0xd8/0x260
[ 990.054483] schedule_timeout+0x13f/0x290
[ 990.059778] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.066149] ? trace_contention_end+0xb3/0x1a0
[ 990.071934] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.077623] ? ww_mutex_lock+0x160/0x160
[ 990.082821] msleep+0x1b/0x30
[ 990.086950] napi_set_threaded+0x3a5/0x440
[ 990.092346] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.097841] netif_set_threaded+0x257/0x400
[ 990.103338] threaded_store+0x1b8/0x220
[ 990.108442] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.113741] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.119037] ? sysfs_kf_write+0x65/0x160
[ 990.124239] ? sysfs_file_ops+0x140/0x140
[ 990.129537] kernfs_fop_write_iter+0x335/0x510
[ 990.135321] ? kernfs_ops+0xf0/0xf0
[ 990.140038] new_sync_write+0x333/0x750
[ 990.145143] ? __lock_acquire+0x508/0xc10
[ 990.150438] ? new_sync_read+0x740/0x740
[ 990.155640] ? lock_acquire.part.0+0xbc/0x260
[ 990.161332] ? ksys_write+0x116/0x250
[ 990.166243] vfs_write+0x657/0x9b0
[ 990.170860] ksys_write+0x116/0x250
[ 990.175575] ? __ia32_sys_read+0xc0/0xc0
[ 990.180774] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.186362] ? rcu_is_watching+0x15/0xd0
[ 990.191562] do_syscall_64+0xf3/0x650
[ 990.196467] ? irq_exit_rcu+0x1a/0x30
[ 990.201376] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.207842] RIP: 0033:0x7f0e39c3022e
[ 990.212650] RSP: 002b:00007ffdb49c8f60 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.221948] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f0e39c3022e
[ 990.230757] RDX: 0000000000000001 RSI: 00005643870c46b0 RDI: 0000000000000007
[ 990.239564] RBP: 00007ffdb49c8f70 R08: 0000000000000000 R09: 0000000000000000
[ 990.248374] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f0e39aa8390
[ 990.257180] R13: 00005643870c46b0 R14: 0000000000000007 R15: 0000000000000003
[ 990.265995]
[ 990.269258] \x0aShowing all locks held in the system:
[ 990.277799] 1 lock held by khungtaskd/185:
[ 990.283191] #0: ffffffff8c6fdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.294676] 4 locks held by python3/3752:
[ 990.299974] #0: ff1100012fef43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.309866] #1: ff1100014ee74a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.320829] #2: ff11000123574b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.331986] #3: ff11000123551d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.342076] 4 locks held by python3/5815:
[ 990.347373] #0: ff1100012fef43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.357264] #1: ff110001ee628e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.368225] #2: ff11000123574b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.379374] #3: ff11000123551d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.389463]
[ 990.391932] =============================================\x0a
[ 1112.596715] INFO: task python3:5815 blocked for more than 245 seconds.
[ 1112.604863] Not tainted 7.1.0-rc4-zuul-g5f6251e63d83 #1
[ 1112.612112] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.621702] task:python3 state:D stack:25672 pid:5815 tgid:5815 ppid:5814 task_flags:0x400100 flags:0x00080000
[ 1112.634905] Call Trace:
[ 1112.638445]
[ 1112.641594] __schedule+0xcc3/0x19c0
[ 1112.646407] ? io_schedule_timeout+0x140/0x140
[ 1112.652193] ? find_held_lock+0x2b/0x80
[ 1112.657297] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.663180] schedule+0xd8/0x260
[ 1112.667587] schedule_preempt_disabled+0x18/0x30
[ 1112.673563] __mutex_lock+0x6af/0x2010
[ 1112.678566] ? threaded_store+0x132/0x220
[ 1112.683856] ? ww_mutex_lock+0x160/0x160
[ 1112.689053] ? selinux_inode_free_security+0x330/0x330
[ 1112.695623] ? cap_capable+0x1d6/0x3e0
[ 1112.700628] ? threaded_store+0x132/0x220
[ 1112.705923] threaded_store+0x132/0x220
[ 1112.711024] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.716319] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.721621] ? sysfs_kf_write+0x65/0x160
[ 1112.726819] ? sysfs_file_ops+0x140/0x140
[ 1112.732112] kernfs_fop_write_iter+0x335/0x510
[ 1112.737895] ? kernfs_ops+0xf0/0xf0
[ 1112.742607] new_sync_write+0x333/0x750
[ 1112.747708] ? __lock_acquire+0x508/0xc10
[ 1112.753005] ? new_sync_read+0x740/0x740
[ 1112.758204] ? lock_acquire.part.0+0xbc/0x260
[ 1112.763891] ? ksys_write+0x116/0x250
[ 1112.768801] vfs_write+0x657/0x9b0
[ 1112.773417] ksys_write+0x116/0x250
[ 1112.778126] ? __ia32_sys_read+0xc0/0xc0
[ 1112.783324] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.788913] ? rcu_is_watching+0x15/0xd0
[ 1112.794114] do_syscall_64+0xf3/0x650
[ 1112.799010] ? trace_hardirqs_off+0xd/0x30
[ 1112.804402] ? exc_page_fault+0xda/0xf0
[ 1112.809505] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.815961] RIP: 0033:0x7f267645d22e
[ 1112.820769] RSP: 002b:00007fffe6448440 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.830064] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f267645d22e
[ 1112.838870] RDX: 0000000000000001 RSI: 0000558cd3c6a6b0 RDI: 0000000000000007
[ 1112.847676] RBP: 00007fffe6448450 R08: 0000000000000000 R09: 0000000000000000
[ 1112.856483] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f26762d5390
[ 1112.865289] R13: 0000558cd3c6a6b0 R14: 0000000000000007 R15: 0000000000000003
[ 1112.874101]
[ 1112.877372] INFO: task python3:5815 is blocked on a mutex likely owned by task python3:3752.
[ 1112.887641] task:python3 state:D stack:25616 pid:3752 tgid:3752 ppid:3751 task_flags:0x400100 flags:0x00080802
[ 1112.900843] Call Trace:
[ 1112.904378]
[ 1112.907526] __schedule+0xcc3/0x19c0
[ 1112.912337] ? io_schedule_timeout+0x140/0x140
[ 1112.918120] ? find_held_lock+0x2b/0x80
[ 1112.923223] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.929398] schedule+0xd8/0x260
[ 1112.933813] schedule_timeout+0x13f/0x290
[ 1112.939109] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.945476] ? trace_contention_end+0xb3/0x1a0
[ 1112.951260] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.956951] ? ww_mutex_lock+0x160/0x160
[ 1112.962149] msleep+0x1b/0x30
[ 1112.966273] napi_set_threaded+0x3a5/0x440
[ 1112.971671] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.977163] netif_set_threaded+0x257/0x400
[ 1112.982655] threaded_store+0x1b8/0x220
[ 1112.987755] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.993051] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.998348] ? sysfs_kf_write+0x65/0x160
[ 1113.003549] ? sysfs_file_ops+0x140/0x140
[ 1113.008844] kernfs_fop_write_iter+0x335/0x510
[ 1113.014629] ? kernfs_ops+0xf0/0xf0
[ 1113.019338] new_sync_write+0x333/0x750
[ 1113.024439] ? __lock_acquire+0x508/0xc10
[ 1113.029733] ? new_sync_read+0x740/0x740
[ 1113.034933] ? lock_acquire.part.0+0xbc/0x260
[ 1113.040619] ? ksys_write+0x116/0x250
[ 1113.045527] vfs_write+0x657/0x9b0
[ 1113.050143] ksys_write+0x116/0x250
[ 1113.054856] ? __ia32_sys_read+0xc0/0xc0
[ 1113.060052] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.065641] ? rcu_is_watching+0x15/0xd0
[ 1113.070839] do_syscall_64+0xf3/0x650
[ 1113.075827] ? irq_exit_rcu+0x1a/0x30
[ 1113.080732] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.087196] RIP: 0033:0x7f0e39c3022e
[ 1113.092002] RSP: 002b:00007ffdb49c8f60 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.101301] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f0e39c3022e
[ 1113.110108] RDX: 0000000000000001 RSI: 00005643870c46b0 RDI: 0000000000000007
[ 1113.118915] RBP: 00007ffdb49c8f70 R08: 0000000000000000 R09: 0000000000000000
[ 1113.127713] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f0e39aa8390
[ 1113.136521] R13: 00005643870c46b0 R14: 0000000000000007 R15: 0000000000000003
[ 1113.145334]
[ 1113.148597] \x0aShowing all locks held in the system:
[ 1113.157134] 1 lock held by khungtaskd/185:
[ 1113.162529] #0: ffffffff8c6fdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.174068] 4 locks held by python3/3752:
[ 1113.179367] #0: ff1100012fef43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.189263] #1: ff1100014ee74a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.189271] #2: ff11000123574b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.189278] #3: ff11000123551d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.221507] 4 locks held by python3/5815:
[ 1113.226803] #0: ff1100012fef43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.236695] #1: ff110001ee628e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.247659] #2: ff11000123574b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.258819] #3: ff11000123551d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.268907] 5 locks held by (systemd-user-)/6869:
[ 1113.274985] #0: ff110017da0543e8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.284876] #1: ff110001658cfa80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.295839] #2: ffffffff8c769b60 (cgroup_mutex){+.+.}-{4:4}, at: cgroup_kn_lock_live+0xe2/0x3e0
[ 1113.306510] #3: ffffffff8c5686f0 (cpu_hotplug_lock){++++}-{0:0}, at: cgroup_procs_write_start+0x3e5/0x650
[ 1113.318152] #4: ffffffff8c7698d0 (cgroup_threadgroup_rwsem){++++}-{0:0}, at: cgroup_procs_write_start+0x3f1/0x650
[ 1113.330582]
[ 1113.333040] =============================================\x0a
[ 1235.478026] INFO: task python3:5815 blocked for more than 368 seconds.
[ 1235.486173] Not tainted 7.1.0-rc4-zuul-g5f6251e63d83 #1
[ 1235.493424] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.503014] task:python3 state:D stack:25672 pid:5815 tgid:5815 ppid:5814 task_flags:0x400100 flags:0x00080002
[ 1235.516220] Call Trace:
[ 1235.519761]
[ 1235.522911] __schedule+0xcc3/0x19c0
[ 1235.527729] ? io_schedule_timeout+0x140/0x140
[ 1235.533517] ? find_held_lock+0x2b/0x80
[ 1235.538624] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.544509] schedule+0xd8/0x260
[ 1235.548926] schedule_preempt_disabled+0x18/0x30
[ 1235.554904] __mutex_lock+0x6af/0x2010
[ 1235.559911] ? threaded_store+0x132/0x220
[ 1235.565211] ? ww_mutex_lock+0x160/0x160
[ 1235.570411] ? selinux_inode_free_security+0x330/0x330
[ 1235.576980] ? cap_capable+0x1d6/0x3e0
[ 1235.581990] ? threaded_store+0x132/0x220
[ 1235.587289] threaded_store+0x132/0x220
[ 1235.592390] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.597682] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.602983] ? sysfs_kf_write+0x65/0x160
[ 1235.608182] ? sysfs_file_ops+0x140/0x140
[ 1235.613479] kernfs_fop_write_iter+0x335/0x510
[ 1235.619257] ? kernfs_ops+0xf0/0xf0
[ 1235.623970] new_sync_write+0x333/0x750
[ 1235.629076] ? __lock_acquire+0x508/0xc10
[ 1235.634373] ? new_sync_read+0x740/0x740
[ 1235.639566] ? lock_acquire.part.0+0xbc/0x260
[ 1235.645254] ? ksys_write+0x116/0x250
[ 1235.650164] vfs_write+0x657/0x9b0
[ 1235.654778] ksys_write+0x116/0x250
[ 1235.659492] ? __ia32_sys_read+0xc0/0xc0
[ 1235.664691] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.670278] ? rcu_is_watching+0x15/0xd0
[ 1235.675480] do_syscall_64+0xf3/0x650
[ 1235.680386] ? trace_hardirqs_off+0xd/0x30
[ 1235.685782] ? exc_page_fault+0xda/0xf0
[ 1235.690877] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.697344] RIP: 0033:0x7f267645d22e
[ 1235.702156] RSP: 002b:00007fffe6448440 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.711457] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f267645d22e
[ 1235.720268] RDX: 0000000000000001 RSI: 0000558cd3c6a6b0 RDI: 0000000000000007
[ 1235.729078] RBP: 00007fffe6448450 R08: 0000000000000000 R09: 0000000000000000
[ 1235.737888] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f26762d5390
[ 1235.746696] R13: 0000558cd3c6a6b0 R14: 0000000000000007 R15: 0000000000000003
[ 1235.755501]
[ 1235.758804] INFO: task python3:5815 is blocked on a mutex likely owned by task python3:3752.
[ 1235.769077] task:python3 state:D stack:25616 pid:3752 tgid:3752 ppid:3751 task_flags:0x400100 flags:0x00080802
[ 1235.782279] Call Trace:
[ 1235.785816]
[ 1235.788969] __schedule+0xcc3/0x19c0
[ 1235.793782] ? io_schedule_timeout+0x140/0x140
[ 1235.799567] ? find_held_lock+0x2b/0x80
[ 1235.804672] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.810850] schedule+0xd8/0x260
[ 1235.815269] schedule_timeout+0x13f/0x290
[ 1235.820558] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.826928] ? trace_contention_end+0xb3/0x1a0
[ 1235.832714] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.838397] ? ww_mutex_lock+0x160/0x160
[ 1235.843598] msleep+0x1b/0x30
[ 1235.847725] napi_set_threaded+0x3a5/0x440
[ 1235.853120] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.858614] netif_set_threaded+0x257/0x400
[ 1235.864109] threaded_store+0x1b8/0x220
[ 1235.869210] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.874508] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.879798] ? sysfs_kf_write+0x65/0x160
[ 1235.884998] ? sysfs_file_ops+0x140/0x140
[ 1235.890295] kernfs_fop_write_iter+0x335/0x510
[ 1235.896079] ? kernfs_ops+0xf0/0xf0
[ 1235.900790] new_sync_write+0x333/0x750
[ 1235.905894] ? __lock_acquire+0x508/0xc10
[ 1235.911191] ? new_sync_read+0x740/0x740
[ 1235.916394] ? lock_acquire.part.0+0xbc/0x260
[ 1235.922081] ? ksys_write+0x116/0x250
[ 1235.926991] vfs_write+0x657/0x9b0
[ 1235.931604] ksys_write+0x116/0x250
[ 1235.936320] ? __ia32_sys_read+0xc0/0xc0
[ 1235.941521] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.947113] ? rcu_is_watching+0x15/0xd0
[ 1235.952312] do_syscall_64+0xf3/0x650
[ 1235.957217] ? irq_exit_rcu+0x1a/0x30
[ 1235.962126] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.968585] RIP: 0033:0x7f0e39c3022e
[ 1235.973394] RSP: 002b:00007ffdb49c8f60 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.982691] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f0e39c3022e
[ 1235.991497] RDX: 0000000000000001 RSI: 00005643870c46b0 RDI: 0000000000000007
[ 1236.000308] RBP: 00007ffdb49c8f70 R08: 0000000000000000 R09: 0000000000000000
[ 1236.009118] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f0e39aa8390
[ 1236.017929] R13: 00005643870c46b0 R14: 0000000000000007 R15: 0000000000000003
[ 1236.026743]
[ 1236.030015] \x0aShowing all locks held in the system:
[ 1236.038557] 1 lock held by khungtaskd/185:
[ 1236.043953] #0: ffffffff8c6fdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.055428] 2 locks held by systemd-journal/803:
[ 1236.061399] #0: ff1100180e2372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1236.071286] #1: ffffffff8c9029a0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1236.084116] 4 locks held by python3/3752:
[ 1236.089410] #0: ff1100012fef43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.099298] #1: ff1100014ee74a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.110262] #2: ff11000123574b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.121418] #3: ff11000123551d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.131512] 4 locks held by python3/5815:
[ 1236.136808] #0: ff1100012fef43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.146697] #1: ff110001ee628e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.157659] #2: ff11000123574b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.168816] #3: ff11000123551d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.178908]
[ 1236.181373] =============================================\x0a
[ 1358.360251] INFO: task python3:5815 blocked for more than 491 seconds.
[ 1358.368401] Not tainted 7.1.0-rc4-zuul-g5f6251e63d83 #1
[ 1358.375653] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.385240] task:python3 state:D stack:25672 pid:5815 tgid:5815 ppid:5814 task_flags:0x400100 flags:0x00080002
[ 1358.398446] Call Trace:
[ 1358.401987]
[ 1358.405135] __schedule+0xcc3/0x19c0
[ 1358.409949] ? io_schedule_timeout+0x140/0x140
[ 1358.415736] ? find_held_lock+0x2b/0x80
[ 1358.420840] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.426725] schedule+0xd8/0x260
[ 1358.431142] schedule_preempt_disabled+0x18/0x30
[ 1358.437120] __mutex_lock+0x6af/0x2010
[ 1358.442123] ? threaded_store+0x132/0x220
[ 1358.447423] ? ww_mutex_lock+0x160/0x160
[ 1358.452609] ? selinux_inode_free_security+0x330/0x330
[ 1358.459181] ? cap_capable+0x1d6/0x3e0
[ 1358.464186] ? threaded_store+0x132/0x220
[ 1358.469555] threaded_store+0x132/0x220
[ 1358.474656] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.479953] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.485250] ? sysfs_kf_write+0x65/0x160
[ 1358.490449] ? sysfs_file_ops+0x140/0x140
[ 1358.495742] kernfs_fop_write_iter+0x335/0x510
[ 1358.501524] ? kernfs_ops+0xf0/0xf0
[ 1358.506234] new_sync_write+0x333/0x750
[ 1358.511339] ? __lock_acquire+0x508/0xc10
[ 1358.516634] ? new_sync_read+0x740/0x740
[ 1358.521835] ? lock_acquire.part.0+0xbc/0x260
[ 1358.527520] ? ksys_write+0x116/0x250
[ 1358.532429] vfs_write+0x657/0x9b0
[ 1358.537042] ksys_write+0x116/0x250
[ 1358.541755] ? __ia32_sys_read+0xc0/0xc0
[ 1358.546951] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.552538] ? rcu_is_watching+0x15/0xd0
[ 1358.557739] do_syscall_64+0xf3/0x650
[ 1358.562634] ? trace_hardirqs_off+0xd/0x30
[ 1358.568029] ? exc_page_fault+0xda/0xf0
[ 1358.573131] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.579597] RIP: 0033:0x7f267645d22e
[ 1358.584406] RSP: 002b:00007fffe6448440 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.593705] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f267645d22e
[ 1358.602511] RDX: 0000000000000001 RSI: 0000558cd3c6a6b0 RDI: 0000000000000007
[ 1358.611317] RBP: 00007fffe6448450 R08: 0000000000000000 R09: 0000000000000000
[ 1358.620124] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f26762d5390
[ 1358.628931] R13: 0000558cd3c6a6b0 R14: 0000000000000007 R15: 0000000000000003
[ 1358.637741]
[ 1358.641047] INFO: task python3:5815 is blocked on a mutex likely owned by task python3:3752.
[ 1358.651320] task:python3 state:D stack:25616 pid:3752 tgid:3752 ppid:3751 task_flags:0x400100 flags:0x00080802
[ 1358.664521] Call Trace:
[ 1358.668061]
[ 1358.671212] __schedule+0xcc3/0x19c0
[ 1358.676015] ? io_schedule_timeout+0x140/0x140
[ 1358.681800] ? find_held_lock+0x2b/0x80
[ 1358.686903] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.693080] schedule+0xd8/0x260
[ 1358.697498] schedule_timeout+0x13f/0x290
[ 1358.702795] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.709163] ? trace_contention_end+0xb3/0x1a0
[ 1358.714947] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.720638] ? ww_mutex_lock+0x160/0x160
[ 1358.725836] msleep+0x1b/0x30
[ 1358.729960] napi_set_threaded+0x3a5/0x440
[ 1358.735355] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.740847] netif_set_threaded+0x257/0x400
[ 1358.746396] threaded_store+0x1b8/0x220
[ 1358.751496] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.756794] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.762082] ? sysfs_kf_write+0x65/0x160
[ 1358.767278] ? sysfs_file_ops+0x140/0x140
[ 1358.772572] kernfs_fop_write_iter+0x335/0x510
[ 1358.778357] ? kernfs_ops+0xf0/0xf0
[ 1358.783126] new_sync_write+0x333/0x750
[ 1358.788227] ? __lock_acquire+0x508/0xc10
[ 1358.793586] ? new_sync_read+0x740/0x740
[ 1358.798783] ? lock_acquire.part.0+0xbc/0x260
[ 1358.804467] ? ksys_write+0x116/0x250
[ 1358.809376] vfs_write+0x657/0x9b0
[ 1358.813990] ksys_write+0x116/0x250
[ 1358.818702] ? __ia32_sys_read+0xc0/0xc0
[ 1358.823901] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.829490] ? rcu_is_watching+0x15/0xd0
[ 1358.834688] do_syscall_64+0xf3/0x650
[ 1358.839595] ? irq_exit_rcu+0x1a/0x30
[ 1358.844495] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.850951] RIP: 0033:0x7f0e39c3022e
[ 1358.855760] RSP: 002b:00007ffdb49c8f60 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.865057] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f0e39c3022e
[ 1358.873861] RDX: 0000000000000001 RSI: 00005643870c46b0 RDI: 0000000000000007
[ 1358.882669] RBP: 00007ffdb49c8f70 R08: 0000000000000000 R09: 0000000000000000
[ 1358.891466] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f0e39aa8390
[ 1358.900274] R13: 00005643870c46b0 R14: 0000000000000007 R15: 0000000000000003
[ 1358.909085]
[ 1358.912356] \x0aShowing all locks held in the system:
[ 1358.920895] 1 lock held by khungtaskd/185:
[ 1358.926291] #0: ffffffff8c6fdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.937773] 4 locks held by python3/3752:
[ 1358.943070] #0: ff1100012fef43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.952960] #1: ff1100014ee74a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.963921] #2: ff11000123574b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.975079] #3: ff11000123551d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.985174] 4 locks held by python3/5815:
[ 1358.990467] #0: ff1100012fef43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1359.000355] #1: ff110001ee628e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1359.011313] #2: ff11000123574b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.022532] #3: ff11000123551d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.032623] 1 lock held by systemd/7678:
[ 1359.037821]
[ 1359.040285] =============================================\x0a