[ 781.626392] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 781.628750] nipa-hw-worker: [5/44] Retrying drivers/net:napi_threaded.py
[ 781.782831] kselftest: Running tests in drivers/net
[ 846.289833] kworker/u130:1 (325) used greatest stack depth: 21456 bytes left
[ 989.652265] INFO: task python3:5674 blocked for more than 122 seconds.
[ 989.660477] Not tainted 7.1.0-rc1-cabx-gaaafcc9812ea #1
[ 989.668169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.677760] task:python3 state:D stack:25672 pid:5674 tgid:5674 ppid:5673 task_flags:0x400100 flags:0x00080000
[ 989.690964] Call Trace:
[ 989.694501]
[ 989.697652] __schedule+0xcba/0x1940
[ 989.702468] ? io_schedule_timeout+0x140/0x140
[ 989.708251] ? find_held_lock+0x2b/0x80
[ 989.713348] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.719230] schedule+0xd4/0x260
[ 989.723649] schedule_preempt_disabled+0x18/0x30
[ 989.729627] __mutex_lock+0x6a3/0x2000
[ 989.734632] ? threaded_store+0x132/0x220
[ 989.739931] ? ww_mutex_lock+0x160/0x160
[ 989.745127] ? selinux_inode_free_security+0x330/0x330
[ 989.751693] ? cap_capable+0x1d6/0x3e0
[ 989.756700] ? threaded_store+0x132/0x220
[ 989.761985] threaded_store+0x132/0x220
[ 989.767084] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.772383] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.777681] ? sysfs_kf_write+0x65/0x160
[ 989.782878] ? sysfs_file_ops+0x140/0x140
[ 989.788172] kernfs_fop_write_iter+0x335/0x510
[ 989.793958] ? kernfs_ops+0xf0/0xf0
[ 989.798667] new_sync_write+0x333/0x750
[ 989.803769] ? __lock_acquire+0x508/0xc10
[ 989.809064] ? new_sync_read+0x740/0x740
[ 989.814264] ? lock_acquire.part.0+0xbc/0x260
[ 989.819949] ? ksys_write+0x116/0x250
[ 989.824857] vfs_write+0x657/0x9b0
[ 989.829472] ksys_write+0x116/0x250
[ 989.834175] ? __ia32_sys_read+0xc0/0xc0
[ 989.839372] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.844959] ? rcu_is_watching+0x15/0xd0
[ 989.850157] do_syscall_64+0xf3/0x650
[ 989.855059] ? trace_hardirqs_off+0xd/0x30
[ 989.860454] ? exc_page_fault+0xda/0xf0
[ 989.865556] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.872015] RIP: 0033:0x7fdb1cb6822e
[ 989.876824] RSP: 002b:00007ffe9cfd9bc0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.886119] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fdb1cb6822e
[ 989.894924] RDX: 0000000000000001 RSI: 000055723bd763d0 RDI: 0000000000000007
[ 989.903731] RBP: 00007ffe9cfd9bd0 R08: 0000000000000000 R09: 0000000000000000
[ 989.912533] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fdb1c9e0390
[ 989.921341] R13: 000055723bd763d0 R14: 0000000000000007 R15: 0000000000000003
[ 989.930154]
[ 989.933439] INFO: task python3:5674 is blocked on a mutex likely owned by task python3:3617.
[ 989.943711] task:python3 state:D stack:25672 pid:3617 tgid:3617 ppid:3616 task_flags:0x400100 flags:0x00080802
[ 989.956914] Call Trace:
[ 989.960449]
[ 989.963597] __schedule+0xcba/0x1940
[ 989.968408] ? io_schedule_timeout+0x140/0x140
[ 989.974192] ? find_held_lock+0x2b/0x80
[ 989.979292] ? __try_to_del_timer_sync+0x155/0x1f0
[ 989.985468] schedule+0xd4/0x260
[ 989.989887] schedule_timeout+0x13f/0x290
[ 989.995182] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.001551] ? trace_contention_end+0xb3/0x1a0
[ 990.007336] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.013014] ? ww_mutex_lock+0x160/0x160
[ 990.018220] msleep+0x1b/0x30
[ 990.022346] napi_set_threaded+0x3a5/0x440
[ 990.027741] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.033233] netif_set_threaded+0x257/0x400
[ 990.038722] threaded_store+0x1b8/0x220
[ 990.043823] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.049119] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.054414] ? sysfs_kf_write+0x65/0x160
[ 990.059610] ? sysfs_file_ops+0x140/0x140
[ 990.064902] kernfs_fop_write_iter+0x335/0x510
[ 990.070685] ? kernfs_ops+0xf0/0xf0
[ 990.075395] new_sync_write+0x333/0x750
[ 990.080494] ? __lock_acquire+0x508/0xc10
[ 990.085789] ? new_sync_read+0x740/0x740
[ 990.090989] ? lock_acquire.part.0+0xbc/0x260
[ 990.096666] ? ksys_write+0x116/0x250
[ 990.101575] vfs_write+0x657/0x9b0
[ 990.106188] ksys_write+0x116/0x250
[ 990.110897] ? __ia32_sys_read+0xc0/0xc0
[ 990.116093] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.121682] ? rcu_is_watching+0x15/0xd0
[ 990.126881] do_syscall_64+0xf3/0x650
[ 990.131784] ? irq_exit_rcu+0x1a/0x30
[ 990.136691] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.143155] RIP: 0033:0x7f1e2e63122e
[ 990.147961] RSP: 002b:00007ffff2c77fa0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.157258] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f1e2e63122e
[ 990.166063] RDX: 0000000000000001 RSI: 0000559ee30433d0 RDI: 0000000000000007
[ 990.174869] RBP: 00007ffff2c77fb0 R08: 0000000000000000 R09: 0000000000000000
[ 990.183677] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f1e2e4a9390
[ 990.192483] R13: 0000559ee30433d0 R14: 0000000000000007 R15: 0000000000000003
[ 990.201294]
[ 990.204552] \x0aShowing all locks held in the system:
[ 990.213087] 1 lock held by khungtaskd/185:
[ 990.218479] #0: ffffffffa8efdf60 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.229961] 4 locks held by python3/3617:
[ 990.235252] #0: ff1100011c6ec3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.245140] #1: ff11000167232c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.256099] #2: ff11000123ff0b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.267261] #3: ff11000124069d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.277345] 4 locks held by python3/5674:
[ 990.282638] #0: ff1100011c6ec3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.292524] #1: ff110001e4e20280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.303482] #2: ff11000123ff0b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.314640] #3: ff11000124069d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.324727]
[ 990.327191] =============================================\x0a
[ 1112.533528] INFO: task python3:5674 blocked for more than 245 seconds.
[ 1112.541678] Not tainted 7.1.0-rc1-cabx-gaaafcc9812ea #1
[ 1112.548929] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.558520] task:python3 state:D stack:25672 pid:5674 tgid:5674 ppid:5673 task_flags:0x400100 flags:0x00080000
[ 1112.571727] Call Trace:
[ 1112.575267]
[ 1112.578418] __schedule+0xcba/0x1940
[ 1112.583233] ? io_schedule_timeout+0x140/0x140
[ 1112.589018] ? find_held_lock+0x2b/0x80
[ 1112.594123] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.600009] schedule+0xd4/0x260
[ 1112.604428] schedule_preempt_disabled+0x18/0x30
[ 1112.610405] __mutex_lock+0x6a3/0x2000
[ 1112.615408] ? threaded_store+0x132/0x220
[ 1112.620707] ? ww_mutex_lock+0x160/0x160
[ 1112.625896] ? selinux_inode_free_security+0x330/0x330
[ 1112.632467] ? cap_capable+0x1d6/0x3e0
[ 1112.637472] ? threaded_store+0x132/0x220
[ 1112.642769] threaded_store+0x132/0x220
[ 1112.647872] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.653169] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.658459] ? sysfs_kf_write+0x65/0x160
[ 1112.663655] ? sysfs_file_ops+0x140/0x140
[ 1112.668950] kernfs_fop_write_iter+0x335/0x510
[ 1112.674726] ? kernfs_ops+0xf0/0xf0
[ 1112.679438] new_sync_write+0x333/0x750
[ 1112.684539] ? __lock_acquire+0x508/0xc10
[ 1112.689904] ? new_sync_read+0x740/0x740
[ 1112.695103] ? lock_acquire.part.0+0xbc/0x260
[ 1112.700780] ? ksys_write+0x116/0x250
[ 1112.705688] vfs_write+0x657/0x9b0
[ 1112.710304] ksys_write+0x116/0x250
[ 1112.715016] ? __ia32_sys_read+0xc0/0xc0
[ 1112.720212] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.725802] ? rcu_is_watching+0x15/0xd0
[ 1112.731000] do_syscall_64+0xf3/0x650
[ 1112.735904] ? trace_hardirqs_off+0xd/0x30
[ 1112.741298] ? exc_page_fault+0xda/0xf0
[ 1112.746400] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.752856] RIP: 0033:0x7fdb1cb6822e
[ 1112.757667] RSP: 002b:00007ffe9cfd9bc0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.766957] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fdb1cb6822e
[ 1112.775766] RDX: 0000000000000001 RSI: 000055723bd763d0 RDI: 0000000000000007
[ 1112.784570] RBP: 00007ffe9cfd9bd0 R08: 0000000000000000 R09: 0000000000000000
[ 1112.793379] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fdb1c9e0390
[ 1112.802185] R13: 000055723bd763d0 R14: 0000000000000007 R15: 0000000000000003
[ 1112.810998]
[ 1112.814310] INFO: task python3:5674 is blocked on a mutex likely owned by task python3:3617.
[ 1112.824579] task:python3 state:D stack:25672 pid:3617 tgid:3617 ppid:3616 task_flags:0x400100 flags:0x00080802
[ 1112.837782] Call Trace:
[ 1112.841320]
[ 1112.844467] __schedule+0xcba/0x1940
[ 1112.849279] ? io_schedule_timeout+0x140/0x140
[ 1112.855064] ? find_held_lock+0x2b/0x80
[ 1112.860165] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.866340] schedule+0xd4/0x260
[ 1112.870758] schedule_timeout+0x13f/0x290
[ 1112.876054] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.882427] ? trace_contention_end+0xb3/0x1a0
[ 1112.888211] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.893903] ? ww_mutex_lock+0x160/0x160
[ 1112.899103] msleep+0x1b/0x30
[ 1112.903229] napi_set_threaded+0x3a5/0x440
[ 1112.908624] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.914118] netif_set_threaded+0x257/0x400
[ 1112.919613] threaded_store+0x1b8/0x220
[ 1112.924717] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.930014] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.935313] ? sysfs_kf_write+0x65/0x160
[ 1112.940514] ? sysfs_file_ops+0x140/0x140
[ 1112.945812] kernfs_fop_write_iter+0x335/0x510
[ 1112.951596] ? kernfs_ops+0xf0/0xf0
[ 1112.956309] new_sync_write+0x333/0x750
[ 1112.961404] ? __lock_acquire+0x508/0xc10
[ 1112.966701] ? new_sync_read+0x740/0x740
[ 1112.971903] ? lock_acquire.part.0+0xbc/0x260
[ 1112.977589] ? ksys_write+0x116/0x250
[ 1112.982499] vfs_write+0x657/0x9b0
[ 1112.987113] ksys_write+0x116/0x250
[ 1112.991826] ? __ia32_sys_read+0xc0/0xc0
[ 1112.997019] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.002609] ? rcu_is_watching+0x15/0xd0
[ 1113.007810] do_syscall_64+0xf3/0x650
[ 1113.012715] ? irq_exit_rcu+0x1a/0x30
[ 1113.017625] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.024091] RIP: 0033:0x7f1e2e63122e
[ 1113.028900] RSP: 002b:00007ffff2c77fa0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.038197] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f1e2e63122e
[ 1113.047006] RDX: 0000000000000001 RSI: 0000559ee30433d0 RDI: 0000000000000007
[ 1113.055816] RBP: 00007ffff2c77fb0 R08: 0000000000000000 R09: 0000000000000000
[ 1113.064626] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f1e2e4a9390
[ 1113.073436] R13: 0000559ee30433d0 R14: 0000000000000007 R15: 0000000000000003
[ 1113.082251]
[ 1113.085519] \x0aShowing all locks held in the system:
[ 1113.094065] 1 lock held by khungtaskd/185:
[ 1113.099461] #0: ffffffffa8efdf60 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.110944] 4 locks held by python3/3617:
[ 1113.116240] #0: ff1100011c6ec3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.126131] #1: ff11000167232c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.137094] #2: ff11000123ff0b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.148251] #3: ff11000124069d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.158340] 4 locks held by python3/5674:
[ 1113.163637] #0: ff1100011c6ec3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.173529] #1: ff110001e4e20280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.184494] #2: ff11000123ff0b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.195652] #3: ff11000124069d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.205745] 5 locks held by systemd/6740:
[ 1113.211039] #0: ff110017d9e943e8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.220931] #1: ff1100014ef18880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.231892] #2: ffffffffa8f69ca0 (cgroup_mutex){+.+.}-{4:4}, at: cgroup_lock_and_drain_offline+0xa6/0x640
[ 1113.243537] #3: ffffffffa8d68970 (cpu_hotplug_lock){++++}-{0:0}, at: cgroup_update_dfl_csses+0x2a3/0x5c0
[ 1113.255075] #4: ffffffffa8f69a10 (cgroup_threadgroup_rwsem){++++}-{0:0}, at: cgroup_update_dfl_csses+0x2af/0x5c0
[ 1113.267405]
[ 1113.269871] =============================================\x0a
[ 1235.413751] INFO: task python3:5674 blocked for more than 368 seconds.
[ 1235.421901] Not tainted 7.1.0-rc1-cabx-gaaafcc9812ea #1
[ 1235.429152] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.438742] task:python3 state:D stack:25672 pid:5674 tgid:5674 ppid:5673 task_flags:0x400100 flags:0x00080002
[ 1235.451947] Call Trace:
[ 1235.455486]
[ 1235.458638] __schedule+0xcba/0x1940
[ 1235.463452] ? io_schedule_timeout+0x140/0x140
[ 1235.469234] ? find_held_lock+0x2b/0x80
[ 1235.474336] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.480221] schedule+0xd4/0x260
[ 1235.484637] schedule_preempt_disabled+0x18/0x30
[ 1235.490613] __mutex_lock+0x6a3/0x2000
[ 1235.495616] ? threaded_store+0x132/0x220
[ 1235.500914] ? ww_mutex_lock+0x160/0x160
[ 1235.506113] ? selinux_inode_free_security+0x330/0x330
[ 1235.512682] ? cap_capable+0x1d6/0x3e0
[ 1235.517687] ? threaded_store+0x132/0x220
[ 1235.522981] threaded_store+0x132/0x220
[ 1235.528082] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.533376] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.538675] ? sysfs_kf_write+0x65/0x160
[ 1235.543866] ? sysfs_file_ops+0x140/0x140
[ 1235.549160] kernfs_fop_write_iter+0x335/0x510
[ 1235.554945] ? kernfs_ops+0xf0/0xf0
[ 1235.559657] new_sync_write+0x333/0x750
[ 1235.564761] ? __lock_acquire+0x508/0xc10
[ 1235.570057] ? new_sync_read+0x740/0x740
[ 1235.575259] ? lock_acquire.part.0+0xbc/0x260
[ 1235.580945] ? ksys_write+0x116/0x250
[ 1235.585854] vfs_write+0x657/0x9b0
[ 1235.590468] ksys_write+0x116/0x250
[ 1235.595180] ? __ia32_sys_read+0xc0/0xc0
[ 1235.600377] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.605964] ? rcu_is_watching+0x15/0xd0
[ 1235.611164] do_syscall_64+0xf3/0x650
[ 1235.616059] ? trace_hardirqs_off+0xd/0x30
[ 1235.621451] ? exc_page_fault+0xda/0xf0
[ 1235.626553] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.633010] RIP: 0033:0x7fdb1cb6822e
[ 1235.637818] RSP: 002b:00007ffe9cfd9bc0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.647113] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fdb1cb6822e
[ 1235.655920] RDX: 0000000000000001 RSI: 000055723bd763d0 RDI: 0000000000000007
[ 1235.664727] RBP: 00007ffe9cfd9bd0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.673533] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fdb1c9e0390
[ 1235.682340] R13: 000055723bd763d0 R14: 0000000000000007 R15: 0000000000000003
[ 1235.691152]
[ 1235.694430] INFO: task python3:5674 is blocked on a mutex likely owned by task python3:3617.
[ 1235.704702] task:python3 state:D stack:25672 pid:3617 tgid:3617 ppid:3616 task_flags:0x400100 flags:0x00080802
[ 1235.717902] Call Trace:
[ 1235.721441]
[ 1235.724588] __schedule+0xcba/0x1940
[ 1235.729404] ? io_schedule_timeout+0x140/0x140
[ 1235.735185] ? find_held_lock+0x2b/0x80
[ 1235.740285] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.746464] schedule+0xd4/0x260
[ 1235.750883] schedule_timeout+0x13f/0x290
[ 1235.756177] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.762542] ? trace_contention_end+0xb3/0x1a0
[ 1235.768326] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.774016] ? ww_mutex_lock+0x160/0x160
[ 1235.779213] msleep+0x1b/0x30
[ 1235.783337] napi_set_threaded+0x3a5/0x440
[ 1235.788731] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.794223] netif_set_threaded+0x257/0x400
[ 1235.799715] threaded_store+0x1b8/0x220
[ 1235.804817] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.810104] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.815400] ? sysfs_kf_write+0x65/0x160
[ 1235.820601] ? sysfs_file_ops+0x140/0x140
[ 1235.825896] kernfs_fop_write_iter+0x335/0x510
[ 1235.831682] ? kernfs_ops+0xf0/0xf0
[ 1235.836386] new_sync_write+0x333/0x750
[ 1235.841477] ? __lock_acquire+0x508/0xc10
[ 1235.846770] ? new_sync_read+0x740/0x740
[ 1235.851971] ? lock_acquire.part.0+0xbc/0x260
[ 1235.857656] ? ksys_write+0x116/0x250
[ 1235.862563] vfs_write+0x657/0x9b0
[ 1235.867178] ksys_write+0x116/0x250
[ 1235.871888] ? __ia32_sys_read+0xc0/0xc0
[ 1235.877085] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.882674] ? rcu_is_watching+0x15/0xd0
[ 1235.887871] do_syscall_64+0xf3/0x650
[ 1235.892775] ? irq_exit_rcu+0x1a/0x30
[ 1235.897683] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.904150] RIP: 0033:0x7f1e2e63122e
[ 1235.908949] RSP: 002b:00007ffff2c77fa0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.918245] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f1e2e63122e
[ 1235.927051] RDX: 0000000000000001 RSI: 0000559ee30433d0 RDI: 0000000000000007
[ 1235.935858] RBP: 00007ffff2c77fb0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.944664] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f1e2e4a9390
[ 1235.953468] R13: 0000559ee30433d0 R14: 0000000000000007 R15: 0000000000000003
[ 1235.962282]
[ 1235.965542] \x0aShowing all locks held in the system:
[ 1235.974082] 1 lock held by khungtaskd/185:
[ 1235.979472] #0: ffffffffa8efdf60 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1235.990954] 4 locks held by python3/3617:
[ 1235.996249] #0: ff1100011c6ec3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.006139] #1: ff11000167232c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.017100] #2: ff11000123ff0b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.028253] #3: ff11000124069d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.038346] 4 locks held by python3/5674:
[ 1236.043641] #0: ff1100011c6ec3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.053529] #1: ff110001e4e20280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.064490] #2: ff11000123ff0b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.075636] #3: ff11000124069d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.085726]
[ 1236.088193] =============================================\x0a
[ 1358.295998] INFO: task python3:5674 blocked for more than 491 seconds.
[ 1358.304157] Not tainted 7.1.0-rc1-cabx-gaaafcc9812ea #1
[ 1358.311409] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.320995] task:python3 state:D stack:25672 pid:5674 tgid:5674 ppid:5673 task_flags:0x400100 flags:0x00080002
[ 1358.334202] Call Trace:
[ 1358.337742]
[ 1358.340889] __schedule+0xcba/0x1940
[ 1358.345697] ? io_schedule_timeout+0x140/0x140
[ 1358.351483] ? find_held_lock+0x2b/0x80
[ 1358.356588] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.362471] schedule+0xd4/0x260
[ 1358.366887] schedule_preempt_disabled+0x18/0x30
[ 1358.372863] __mutex_lock+0x6a3/0x2000
[ 1358.377865] ? threaded_store+0x132/0x220
[ 1358.383163] ? ww_mutex_lock+0x160/0x160
[ 1358.388362] ? selinux_inode_free_security+0x330/0x330
[ 1358.394930] ? cap_capable+0x1d6/0x3e0
[ 1358.399940] ? threaded_store+0x132/0x220
[ 1358.405237] threaded_store+0x132/0x220
[ 1358.410337] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.415635] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.420932] ? sysfs_kf_write+0x65/0x160
[ 1358.426133] ? sysfs_file_ops+0x140/0x140
[ 1358.431428] kernfs_fop_write_iter+0x335/0x510
[ 1358.437211] ? kernfs_ops+0xf0/0xf0
[ 1358.441923] new_sync_write+0x333/0x750
[ 1358.447023] ? __lock_acquire+0x508/0xc10
[ 1358.452318] ? new_sync_read+0x740/0x740
[ 1358.457519] ? lock_acquire.part.0+0xbc/0x260
[ 1358.463207] ? ksys_write+0x116/0x250
[ 1358.468116] vfs_write+0x657/0x9b0
[ 1358.472728] ksys_write+0x116/0x250
[ 1358.477440] ? __ia32_sys_read+0xc0/0xc0
[ 1358.482637] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.488227] ? rcu_is_watching+0x15/0xd0
[ 1358.493427] do_syscall_64+0xf3/0x650
[ 1358.498330] ? trace_hardirqs_off+0xd/0x30
[ 1358.503724] ? exc_page_fault+0xda/0xf0
[ 1358.508827] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.515292] RIP: 0033:0x7fdb1cb6822e
[ 1358.520098] RSP: 002b:00007ffe9cfd9bc0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.529398] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fdb1cb6822e
[ 1358.538207] RDX: 0000000000000001 RSI: 000055723bd763d0 RDI: 0000000000000007
[ 1358.547013] RBP: 00007ffe9cfd9bd0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.555817] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fdb1c9e0390
[ 1358.564622] R13: 000055723bd763d0 R14: 0000000000000007 R15: 0000000000000003
[ 1358.573432]
[ 1358.576729] INFO: task python3:5674 is blocked on a mutex likely owned by task python3:3617.
[ 1358.587000] task:python3 state:D stack:25672 pid:3617 tgid:3617 ppid:3616 task_flags:0x400100 flags:0x00080802
[ 1358.600200] Call Trace:
[ 1358.603738]
[ 1358.606886] __schedule+0xcba/0x1940
[ 1358.611694] ? io_schedule_timeout+0x140/0x140
[ 1358.617478] ? find_held_lock+0x2b/0x80
[ 1358.622581] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.628757] schedule+0xd4/0x260
[ 1358.633174] schedule_timeout+0x13f/0x290
[ 1358.638472] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.644845] ? trace_contention_end+0xb3/0x1a0
[ 1358.650626] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.656313] ? ww_mutex_lock+0x160/0x160
[ 1358.661509] msleep+0x1b/0x30
[ 1358.665635] napi_set_threaded+0x3a5/0x440
[ 1358.671028] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.676521] netif_set_threaded+0x257/0x400
[ 1358.682006] threaded_store+0x1b8/0x220
[ 1358.687106] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.692404] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.697698] ? sysfs_kf_write+0x65/0x160
[ 1358.702898] ? sysfs_file_ops+0x140/0x140
[ 1358.708185] kernfs_fop_write_iter+0x335/0x510
[ 1358.713968] ? kernfs_ops+0xf0/0xf0
[ 1358.718680] new_sync_write+0x333/0x750
[ 1358.723782] ? __lock_acquire+0x508/0xc10
[ 1358.729080] ? new_sync_read+0x740/0x740
[ 1358.734276] ? lock_acquire.part.0+0xbc/0x260
[ 1358.739960] ? ksys_write+0x116/0x250
[ 1358.744869] vfs_write+0x657/0x9b0
[ 1358.749482] ksys_write+0x116/0x250
[ 1358.754191] ? __ia32_sys_read+0xc0/0xc0
[ 1358.759389] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.764978] ? rcu_is_watching+0x15/0xd0
[ 1358.770178] do_syscall_64+0xf3/0x650
[ 1358.775084] ? irq_exit_rcu+0x1a/0x30
[ 1358.779980] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.786446] RIP: 0033:0x7f1e2e63122e
[ 1358.791250] RSP: 002b:00007ffff2c77fa0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.800549] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f1e2e63122e
[ 1358.809354] RDX: 0000000000000001 RSI: 0000559ee30433d0 RDI: 0000000000000007
[ 1358.818160] RBP: 00007ffff2c77fb0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.826966] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f1e2e4a9390
[ 1358.835772] R13: 0000559ee30433d0 R14: 0000000000000007 R15: 0000000000000003
[ 1358.844582]
[ 1358.847855] \x0aShowing all locks held in the system:
[ 1358.856393] 1 lock held by khungtaskd/185:
[ 1358.861785] #0: ffffffffa8efdf60 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.873266] 4 locks held by python3/3617:
[ 1358.878559] #0: ff1100011c6ec3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.888447] #1: ff11000167232c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.899409] #2: ff11000123ff0b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.910565] #3: ff11000124069d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.920660] 4 locks held by python3/5674:
[ 1358.925951] #0: ff1100011c6ec3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.935845] #1: ff110001e4e20280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.946807] #2: ff11000123ff0b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.957961] #3: ff11000124069d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.968051]
[ 1358.970516] =============================================\x0a