[ 785.222159] nipa-hw-worker: drivers/net:napi_threaded.py: timed out [ 785.224739] nipa-hw-worker: [5/47] Retrying drivers/net:napi_threaded.py [ 785.379519] kselftest: Running tests in drivers/net [ 989.691873] INFO: task python3:5702 blocked for more than 122 seconds. [ 989.700102] Not tainted 7.1.0-rc3-xlsl-gd1cd808a79f4 #1 [ 989.708058] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 989.717646] task:python3 state:D stack:25672 pid:5702 tgid:5702 ppid:5701 task_flags:0x400100 flags:0x00080000 [ 989.730851] Call Trace: [ 989.734388] [ 989.737537] __schedule+0xcc3/0x19c0 [ 989.742355] ? io_schedule_timeout+0x140/0x140 [ 989.748138] ? find_held_lock+0x2b/0x80 [ 989.753234] ? __lock_release.isra.0+0x6b/0x1a0 [ 989.759120] schedule+0xd8/0x260 [ 989.763536] schedule_preempt_disabled+0x18/0x30 [ 989.769515] __mutex_lock+0x6af/0x2010 [ 989.774516] ? threaded_store+0x132/0x220 [ 989.779813] ? ww_mutex_lock+0x160/0x160 [ 989.785010] ? selinux_inode_free_security+0x330/0x330 [ 989.791580] ? cap_capable+0x1d6/0x3e0 [ 989.796589] ? threaded_store+0x132/0x220 [ 989.801887] threaded_store+0x132/0x220 [ 989.806988] ? xps_rxqs_store+0x2e0/0x2e0 [ 989.812286] ? sysfs_file_kobj+0xbd/0x1e0 [ 989.817583] ? sysfs_kf_write+0x65/0x160 [ 989.822782] ? sysfs_file_ops+0x140/0x140 [ 989.828078] kernfs_fop_write_iter+0x335/0x510 [ 989.833864] ? kernfs_ops+0xf0/0xf0 [ 989.838574] new_sync_write+0x333/0x750 [ 989.843678] ? __lock_acquire+0x508/0xc10 [ 989.848973] ? new_sync_read+0x740/0x740 [ 989.854170] ? lock_acquire.part.0+0xbc/0x260 [ 989.859855] ? ksys_write+0x116/0x250 [ 989.864765] vfs_write+0x657/0x9b0 [ 989.869381] ksys_write+0x116/0x250 [ 989.874093] ? __ia32_sys_read+0xc0/0xc0 [ 989.879280] ? kernfs_fop_llseek+0x160/0x1f0 [ 989.884866] ? rcu_is_watching+0x15/0xd0 [ 989.890066] do_syscall_64+0xf3/0x650 [ 989.894969] ? trace_hardirqs_off+0xd/0x30 [ 989.900362] ? exc_page_fault+0xda/0xf0 [ 989.905464] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 989.911929] RIP: 0033:0x7fc322e7c22e [ 989.916748] RSP: 002b:00007ffd2d4c7fa0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 989.926038] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc322e7c22e [ 989.934843] RDX: 0000000000000001 RSI: 0000563182516cd0 RDI: 0000000000000007 [ 989.943650] RBP: 00007ffd2d4c7fb0 R08: 0000000000000000 R09: 0000000000000000 [ 989.952455] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc322cf4390 [ 989.961260] R13: 0000563182516cd0 R14: 0000000000000007 R15: 0000000000000003 [ 989.970071] [ 989.973381] INFO: task python3:5702 is blocked on a mutex likely owned by task python3:3674. [ 989.983642] task:python3 state:D stack:25672 pid:3674 tgid:3674 ppid:3673 task_flags:0x400100 flags:0x00080802 [ 989.996843] Call Trace: [ 990.000379] [ 990.003527] __schedule+0xcc3/0x19c0 [ 990.008339] ? io_schedule_timeout+0x140/0x140 [ 990.014123] ? find_held_lock+0x2b/0x80 [ 990.019225] ? __try_to_del_timer_sync+0x155/0x1f0 [ 990.025403] schedule+0xd8/0x260 [ 990.029819] schedule_timeout+0x13f/0x290 [ 990.035116] ? hrtimer_nanosleep_restart+0x270/0x270 [ 990.041482] ? trace_contention_end+0xb3/0x1a0 [ 990.047267] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 990.052954] ? ww_mutex_lock+0x160/0x160 [ 990.058152] msleep+0x1b/0x30 [ 990.062278] napi_set_threaded+0x3a5/0x440 [ 990.067672] ? napi_resume_irqs+0x1a0/0x1a0 [ 990.073164] netif_set_threaded+0x257/0x400 [ 990.078649] threaded_store+0x1b8/0x220 [ 990.083748] ? xps_rxqs_store+0x2e0/0x2e0 [ 990.089043] ? sysfs_file_kobj+0xbd/0x1e0 [ 990.094337] ? sysfs_kf_write+0x65/0x160 [ 990.099533] ? sysfs_file_ops+0x140/0x140 [ 990.104829] kernfs_fop_write_iter+0x335/0x510 [ 990.110612] ? kernfs_ops+0xf0/0xf0 [ 990.115322] new_sync_write+0x333/0x750 [ 990.120421] ? __lock_acquire+0x508/0xc10 [ 990.125721] ? new_sync_read+0x740/0x740 [ 990.130919] ? lock_acquire.part.0+0xbc/0x260 [ 990.136602] ? ksys_write+0x116/0x250 [ 990.141512] vfs_write+0x657/0x9b0 [ 990.146126] ksys_write+0x116/0x250 [ 990.150838] ? __ia32_sys_read+0xc0/0xc0 [ 990.156037] ? kernfs_fop_llseek+0x160/0x1f0 [ 990.161624] ? rcu_is_watching+0x15/0xd0 [ 990.166823] do_syscall_64+0xf3/0x650 [ 990.171729] ? irq_exit_rcu+0x1a/0x30 [ 990.176628] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 990.183093] RIP: 0033:0x7faa1da2922e [ 990.187902] RSP: 002b:00007fff295935f0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 990.197196] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007faa1da2922e [ 990.206004] RDX: 0000000000000001 RSI: 000055cc23510cd0 RDI: 0000000000000007 [ 990.214811] RBP: 00007fff29593600 R08: 0000000000000000 R09: 0000000000000000 [ 990.223618] R10: 0000000000000000 R11: 0000000000000202 R12: 00007faa1d8a1390 [ 990.232417] R13: 000055cc23510cd0 R14: 0000000000000007 R15: 0000000000000003 [ 990.241219] [ 990.244478] \x0aShowing all locks held in the system: [ 990.253015] 1 lock held by khungtaskd/185: [ 990.258407] #0: ffffffffae6fdf60 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 990.269887] 4 locks held by python3/3674: [ 990.275181] #0: ff1100012f4113e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.285068] #1: ff110001685ba680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.296029] #2: ff110001225beb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.307185] #3: ff110001225a1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.317285] 4 locks held by python3/5702: [ 990.322580] #0: ff1100012f4113e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.332472] #1: ff110001ec723e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.343445] #2: ff110001225beb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.354604] #3: ff110001225a1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.364692] [ 990.367159] =============================================\x0a [ 1112.573125] INFO: task python3:5702 blocked for more than 245 seconds. [ 1112.581275] Not tainted 7.1.0-rc3-xlsl-gd1cd808a79f4 #1 [ 1112.588527] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1112.598117] task:python3 state:D stack:25672 pid:5702 tgid:5702 ppid:5701 task_flags:0x400100 flags:0x00080000 [ 1112.611325] Call Trace: [ 1112.614864] [ 1112.618014] __schedule+0xcc3/0x19c0 [ 1112.622834] ? io_schedule_timeout+0x140/0x140 [ 1112.628619] ? find_held_lock+0x2b/0x80 [ 1112.633728] ? __lock_release.isra.0+0x6b/0x1a0 [ 1112.639614] schedule+0xd8/0x260 [ 1112.644033] schedule_preempt_disabled+0x18/0x30 [ 1112.650006] __mutex_lock+0x6af/0x2010 [ 1112.655013] ? threaded_store+0x132/0x220 [ 1112.660312] ? ww_mutex_lock+0x160/0x160 [ 1112.665512] ? selinux_inode_free_security+0x330/0x330 [ 1112.672082] ? cap_capable+0x1d6/0x3e0 [ 1112.677091] ? threaded_store+0x132/0x220 [ 1112.682389] threaded_store+0x132/0x220 [ 1112.687494] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.692792] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.698092] ? sysfs_kf_write+0x65/0x160 [ 1112.703296] ? sysfs_file_ops+0x140/0x140 [ 1112.708593] kernfs_fop_write_iter+0x335/0x510 [ 1112.714371] ? kernfs_ops+0xf0/0xf0 [ 1112.719083] new_sync_write+0x333/0x750 [ 1112.724187] ? __lock_acquire+0x508/0xc10 [ 1112.729482] ? new_sync_read+0x740/0x740 [ 1112.734684] ? lock_acquire.part.0+0xbc/0x260 [ 1112.740371] ? ksys_write+0x116/0x250 [ 1112.745276] vfs_write+0x657/0x9b0 [ 1112.749892] ksys_write+0x116/0x250 [ 1112.754603] ? __ia32_sys_read+0xc0/0xc0 [ 1112.759806] ? kernfs_fop_llseek+0x160/0x1f0 [ 1112.765395] ? rcu_is_watching+0x15/0xd0 [ 1112.770600] do_syscall_64+0xf3/0x650 [ 1112.775509] ? trace_hardirqs_off+0xd/0x30 [ 1112.780905] ? exc_page_fault+0xda/0xf0 [ 1112.785999] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1112.792466] RIP: 0033:0x7fc322e7c22e [ 1112.797276] RSP: 002b:00007ffd2d4c7fa0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1112.806565] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc322e7c22e [ 1112.815375] RDX: 0000000000000001 RSI: 0000563182516cd0 RDI: 0000000000000007 [ 1112.824182] RBP: 00007ffd2d4c7fb0 R08: 0000000000000000 R09: 0000000000000000 [ 1112.832985] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc322cf4390 [ 1112.841796] R13: 0000563182516cd0 R14: 0000000000000007 R15: 0000000000000003 [ 1112.850611] [ 1112.853925] INFO: task python3:5702 is blocked on a mutex likely owned by task python3:3674. [ 1112.864205] task:python3 state:D stack:25672 pid:3674 tgid:3674 ppid:3673 task_flags:0x400100 flags:0x00080802 [ 1112.877410] Call Trace: [ 1112.880948] [ 1112.884101] __schedule+0xcc3/0x19c0 [ 1112.888916] ? io_schedule_timeout+0x140/0x140 [ 1112.894703] ? find_held_lock+0x2b/0x80 [ 1112.899808] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1112.905990] schedule+0xd8/0x260 [ 1112.910410] schedule_timeout+0x13f/0x290 [ 1112.915710] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1112.922078] ? trace_contention_end+0xb3/0x1a0 [ 1112.927865] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1112.933555] ? ww_mutex_lock+0x160/0x160 [ 1112.938755] msleep+0x1b/0x30 [ 1112.942882] napi_set_threaded+0x3a5/0x440 [ 1112.948277] ? napi_resume_irqs+0x1a0/0x1a0 [ 1112.953774] netif_set_threaded+0x257/0x400 [ 1112.959267] threaded_store+0x1b8/0x220 [ 1112.964371] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.969668] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.974969] ? sysfs_kf_write+0x65/0x160 [ 1112.980166] ? sysfs_file_ops+0x140/0x140 [ 1112.985463] kernfs_fop_write_iter+0x335/0x510 [ 1112.991248] ? kernfs_ops+0xf0/0xf0 [ 1112.995962] new_sync_write+0x333/0x750 [ 1113.001068] ? __lock_acquire+0x508/0xc10 [ 1113.006365] ? new_sync_read+0x740/0x740 [ 1113.011566] ? lock_acquire.part.0+0xbc/0x260 [ 1113.017255] ? ksys_write+0x116/0x250 [ 1113.022167] vfs_write+0x657/0x9b0 [ 1113.026781] ksys_write+0x116/0x250 [ 1113.031497] ? __ia32_sys_read+0xc0/0xc0 [ 1113.036695] ? kernfs_fop_llseek+0x160/0x1f0 [ 1113.042284] ? rcu_is_watching+0x15/0xd0 [ 1113.047486] do_syscall_64+0xf3/0x650 [ 1113.052393] ? irq_exit_rcu+0x1a/0x30 [ 1113.057302] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1113.063770] RIP: 0033:0x7faa1da2922e [ 1113.068580] RSP: 002b:00007fff295935f0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1113.077881] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007faa1da2922e [ 1113.086692] RDX: 0000000000000001 RSI: 000055cc23510cd0 RDI: 0000000000000007 [ 1113.095504] RBP: 00007fff29593600 R08: 0000000000000000 R09: 0000000000000000 [ 1113.104311] R10: 0000000000000000 R11: 0000000000000202 R12: 00007faa1d8a1390 [ 1113.113126] R13: 000055cc23510cd0 R14: 0000000000000007 R15: 0000000000000003 [ 1113.121946] [ 1113.125208] \x0aShowing all locks held in the system: [ 1113.133746] 1 lock held by khungtaskd/185: [ 1113.139139] #0: ffffffffae6fdf60 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1113.150617] 2 locks held by systemd-journal/803: [ 1113.156598] #0: ff1100180e2b72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0 [ 1113.166489] #1: ffffffffae902b20 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120 [ 1113.179318] 4 locks held by python3/3674: [ 1113.184613] #0: ff1100012f4113e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.194504] #1: ff110001685ba680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.205464] #2: ff110001225beb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.216622] #3: ff110001225a1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.226716] 4 locks held by python3/5702: [ 1113.232015] #0: ff1100012f4113e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.241895] #1: ff110001ec723e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.252859] #2: ff110001225beb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.264016] #3: ff110001225a1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.274098] [ 1113.276571] =============================================\x0a [ 1235.454371] INFO: task python3:5702 blocked for more than 368 seconds. [ 1235.462522] Not tainted 7.1.0-rc3-xlsl-gd1cd808a79f4 #1 [ 1235.469775] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1235.479367] task:python3 state:D stack:25672 pid:5702 tgid:5702 ppid:5701 task_flags:0x400100 flags:0x00080002 [ 1235.492572] Call Trace: [ 1235.496114] [ 1235.499266] __schedule+0xcc3/0x19c0 [ 1235.504085] ? io_schedule_timeout+0x140/0x140 [ 1235.509872] ? find_held_lock+0x2b/0x80 [ 1235.514979] ? __lock_release.isra.0+0x6b/0x1a0 [ 1235.520863] schedule+0xd8/0x260 [ 1235.525280] schedule_preempt_disabled+0x18/0x30 [ 1235.531262] __mutex_lock+0x6af/0x2010 [ 1235.536265] ? threaded_store+0x132/0x220 [ 1235.541567] ? ww_mutex_lock+0x160/0x160 [ 1235.546766] ? selinux_inode_free_security+0x330/0x330 [ 1235.553334] ? cap_capable+0x1d6/0x3e0 [ 1235.558344] ? threaded_store+0x132/0x220 [ 1235.563643] threaded_store+0x132/0x220 [ 1235.568743] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.574040] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.579331] ? sysfs_kf_write+0x65/0x160 [ 1235.584532] ? sysfs_file_ops+0x140/0x140 [ 1235.589828] kernfs_fop_write_iter+0x335/0x510 [ 1235.595615] ? kernfs_ops+0xf0/0xf0 [ 1235.600325] new_sync_write+0x333/0x750 [ 1235.605420] ? __lock_acquire+0x508/0xc10 [ 1235.610716] ? new_sync_read+0x740/0x740 [ 1235.615920] ? lock_acquire.part.0+0xbc/0x260 [ 1235.621606] ? ksys_write+0x116/0x250 [ 1235.626516] vfs_write+0x657/0x9b0 [ 1235.631132] ksys_write+0x116/0x250 [ 1235.635844] ? __ia32_sys_read+0xc0/0xc0 [ 1235.641033] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.646624] ? rcu_is_watching+0x15/0xd0 [ 1235.651826] do_syscall_64+0xf3/0x650 [ 1235.656734] ? trace_hardirqs_off+0xd/0x30 [ 1235.662129] ? exc_page_fault+0xda/0xf0 [ 1235.667231] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.673688] RIP: 0033:0x7fc322e7c22e [ 1235.678500] RSP: 002b:00007ffd2d4c7fa0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.687800] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc322e7c22e [ 1235.696606] RDX: 0000000000000001 RSI: 0000563182516cd0 RDI: 0000000000000007 [ 1235.705416] RBP: 00007ffd2d4c7fb0 R08: 0000000000000000 R09: 0000000000000000 [ 1235.714225] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc322cf4390 [ 1235.723035] R13: 0000563182516cd0 R14: 0000000000000007 R15: 0000000000000003 [ 1235.731849] [ 1235.735125] INFO: task python3:5702 is blocked on a mutex likely owned by task python3:3674. [ 1235.745399] task:python3 state:D stack:25672 pid:3674 tgid:3674 ppid:3673 task_flags:0x400100 flags:0x00080802 [ 1235.758603] Call Trace: [ 1235.762141] [ 1235.765293] __schedule+0xcc3/0x19c0 [ 1235.770105] ? io_schedule_timeout+0x140/0x140 [ 1235.775890] ? find_held_lock+0x2b/0x80 [ 1235.780986] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1235.787163] schedule+0xd8/0x260 [ 1235.791582] schedule_timeout+0x13f/0x290 [ 1235.796872] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1235.803234] ? trace_contention_end+0xb3/0x1a0 [ 1235.809018] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1235.814709] ? ww_mutex_lock+0x160/0x160 [ 1235.819910] msleep+0x1b/0x30 [ 1235.824035] napi_set_threaded+0x3a5/0x440 [ 1235.829430] ? napi_resume_irqs+0x1a0/0x1a0 [ 1235.834923] netif_set_threaded+0x257/0x400 [ 1235.840418] threaded_store+0x1b8/0x220 [ 1235.845520] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.850818] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.856113] ? sysfs_kf_write+0x65/0x160 [ 1235.861313] ? sysfs_file_ops+0x140/0x140 [ 1235.866601] kernfs_fop_write_iter+0x335/0x510 [ 1235.872384] ? kernfs_ops+0xf0/0xf0 [ 1235.877095] new_sync_write+0x333/0x750 [ 1235.882199] ? __lock_acquire+0x508/0xc10 [ 1235.887495] ? new_sync_read+0x740/0x740 [ 1235.892696] ? lock_acquire.part.0+0xbc/0x260 [ 1235.898382] ? ksys_write+0x116/0x250 [ 1235.903293] vfs_write+0x657/0x9b0 [ 1235.907908] ksys_write+0x116/0x250 [ 1235.912622] ? __ia32_sys_read+0xc0/0xc0 [ 1235.917819] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.923407] ? rcu_is_watching+0x15/0xd0 [ 1235.928607] do_syscall_64+0xf3/0x650 [ 1235.933514] ? irq_exit_rcu+0x1a/0x30 [ 1235.938415] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.944885] RIP: 0033:0x7faa1da2922e [ 1235.949693] RSP: 002b:00007fff295935f0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.958991] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007faa1da2922e [ 1235.967799] RDX: 0000000000000001 RSI: 000055cc23510cd0 RDI: 0000000000000007 [ 1235.976610] RBP: 00007fff29593600 R08: 0000000000000000 R09: 0000000000000000 [ 1235.985419] R10: 0000000000000000 R11: 0000000000000202 R12: 00007faa1d8a1390 [ 1235.994229] R13: 000055cc23510cd0 R14: 0000000000000007 R15: 0000000000000003 [ 1236.003044] [ 1236.006310] \x0aShowing all locks held in the system: [ 1236.014840] 1 lock held by khungtaskd/185: [ 1236.020236] #0: ffffffffae6fdf60 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1236.031698] 2 locks held by systemd-journal/803: [ 1236.037678] #0: ff1100180e4372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0 [ 1236.047568] #1: ffffffffae902b20 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120 [ 1236.060395] 4 locks held by python3/3674: [ 1236.065691] #0: ff1100012f4113e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.075582] #1: ff110001685ba680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.086544] #2: ff110001225beb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.097704] #3: ff110001225a1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.107799] 4 locks held by python3/5702: [ 1236.113097] #0: ff1100012f4113e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.122987] #1: ff110001ec723e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.133950] #2: ff110001225beb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.145106] #3: ff110001225a1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.155198] [ 1236.157666] =============================================\x0a [ 1255.934656] kworker/u129:11 (291) used greatest stack depth: 23024 bytes left [ 1358.335626] INFO: task python3:5702 blocked for more than 491 seconds. [ 1358.343780] Not tainted 7.1.0-rc3-xlsl-gd1cd808a79f4 #1 [ 1358.351032] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1358.360624] task:python3 state:D stack:25672 pid:5702 tgid:5702 ppid:5701 task_flags:0x400100 flags:0x00080002 [ 1358.373827] Call Trace: [ 1358.377365] [ 1358.380514] __schedule+0xcc3/0x19c0 [ 1358.385330] ? io_schedule_timeout+0x140/0x140 [ 1358.391114] ? find_held_lock+0x2b/0x80 [ 1358.396218] ? __lock_release.isra.0+0x6b/0x1a0 [ 1358.402101] schedule+0xd8/0x260 [ 1358.406516] schedule_preempt_disabled+0x18/0x30 [ 1358.412493] __mutex_lock+0x6af/0x2010 [ 1358.417495] ? threaded_store+0x132/0x220 [ 1358.422796] ? ww_mutex_lock+0x160/0x160 [ 1358.427993] ? selinux_inode_free_security+0x330/0x330 [ 1358.434552] ? cap_capable+0x1d6/0x3e0 [ 1358.439633] ? threaded_store+0x132/0x220 [ 1358.444928] threaded_store+0x132/0x220 [ 1358.450031] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.455327] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.460617] ? sysfs_kf_write+0x65/0x160 [ 1358.465818] ? sysfs_file_ops+0x140/0x140 [ 1358.471113] kernfs_fop_write_iter+0x335/0x510 [ 1358.476898] ? kernfs_ops+0xf0/0xf0 [ 1358.481609] new_sync_write+0x333/0x750 [ 1358.486711] ? __lock_acquire+0x508/0xc10 [ 1358.492006] ? new_sync_read+0x740/0x740 [ 1358.497207] ? lock_acquire.part.0+0xbc/0x260 [ 1358.502892] ? ksys_write+0x116/0x250 [ 1358.507803] vfs_write+0x657/0x9b0 [ 1358.512418] ksys_write+0x116/0x250 [ 1358.517121] ? __ia32_sys_read+0xc0/0xc0 [ 1358.522321] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.527910] ? rcu_is_watching+0x15/0xd0 [ 1358.533110] do_syscall_64+0xf3/0x650 [ 1358.538013] ? trace_hardirqs_off+0xd/0x30 [ 1358.543406] ? exc_page_fault+0xda/0xf0 [ 1358.548510] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.554978] RIP: 0033:0x7fc322e7c22e [ 1358.559789] RSP: 002b:00007ffd2d4c7fa0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.569088] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc322e7c22e [ 1358.577899] RDX: 0000000000000001 RSI: 0000563182516cd0 RDI: 0000000000000007 [ 1358.586711] RBP: 00007ffd2d4c7fb0 R08: 0000000000000000 R09: 0000000000000000 [ 1358.586714] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc322cf4390 [ 1358.586716] R13: 0000563182516cd0 R14: 0000000000000007 R15: 0000000000000003 [ 1358.586723] [ 1358.586785] INFO: task python3:5702 is blocked on a mutex likely owned by task python3:3674. [ 1358.626712] task:python3 state:D stack:25672 pid:3674 tgid:3674 ppid:3673 task_flags:0x400100 flags:0x00080802 [ 1358.639915] Call Trace: [ 1358.643454] [ 1358.646606] __schedule+0xcc3/0x19c0 [ 1358.651418] ? io_schedule_timeout+0x140/0x140 [ 1358.657202] ? find_held_lock+0x2b/0x80 [ 1358.662306] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1358.668485] schedule+0xd8/0x260 [ 1358.672902] schedule_timeout+0x13f/0x290 [ 1358.678201] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1358.684569] ? trace_contention_end+0xb3/0x1a0 [ 1358.690356] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1358.696046] ? ww_mutex_lock+0x160/0x160 [ 1358.701244] msleep+0x1b/0x30 [ 1358.705369] napi_set_threaded+0x3a5/0x440 [ 1358.710766] ? napi_resume_irqs+0x1a0/0x1a0 [ 1358.716262] netif_set_threaded+0x257/0x400 [ 1358.721754] threaded_store+0x1b8/0x220 [ 1358.726860] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.732156] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.737454] ? sysfs_kf_write+0x65/0x160 [ 1358.742654] ? sysfs_file_ops+0x140/0x140 [ 1358.747950] kernfs_fop_write_iter+0x335/0x510 [ 1358.753736] ? kernfs_ops+0xf0/0xf0 [ 1358.758448] new_sync_write+0x333/0x750 [ 1358.763549] ? __lock_acquire+0x508/0xc10 [ 1358.768848] ? new_sync_read+0x740/0x740 [ 1358.774047] ? lock_acquire.part.0+0xbc/0x260 [ 1358.779733] ? ksys_write+0x116/0x250 [ 1358.784644] vfs_write+0x657/0x9b0 [ 1358.789261] ksys_write+0x116/0x250 [ 1358.793974] ? __ia32_sys_read+0xc0/0xc0 [ 1358.799175] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.804765] ? rcu_is_watching+0x15/0xd0 [ 1358.809957] do_syscall_64+0xf3/0x650 [ 1358.814866] ? irq_exit_rcu+0x1a/0x30 [ 1358.819773] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.826240] RIP: 0033:0x7faa1da2922e [ 1358.831048] RSP: 002b:00007fff295935f0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.840345] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007faa1da2922e [ 1358.849145] RDX: 0000000000000001 RSI: 000055cc23510cd0 RDI: 0000000000000007 [ 1358.857956] RBP: 00007fff29593600 R08: 0000000000000000 R09: 0000000000000000 [ 1358.866765] R10: 0000000000000000 R11: 0000000000000202 R12: 00007faa1d8a1390 [ 1358.875574] R13: 000055cc23510cd0 R14: 0000000000000007 R15: 0000000000000003 [ 1358.884387] [ 1358.887658] \x0aShowing all locks held in the system: [ 1358.896197] 1 lock held by khungtaskd/185: [ 1358.901591] #0: ffffffffae6fdf60 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1358.913074] 4 locks held by python3/3674: [ 1358.918363] #0: ff1100012f4113e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1358.928246] #1: ff110001685ba680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1358.939206] #2: ff110001225beb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1358.950361] #3: ff110001225a1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1358.960456] 4 locks held by python3/5702: [ 1358.965751] #0: ff1100012f4113e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1358.975639] #1: ff110001ec723e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1358.986594] #2: ff110001225beb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1358.997755] #3: ff110001225a1d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1359.007849] 5 locks held by (systemd-user-)/7547: [ 1359.013924] #0: ff1100012f5013e8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1359.023812] #1: ff1100015a1d4480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1359.034773] #2: ffffffffae769ce0 (cgroup_mutex){+.+.}-{4:4}, at: cgroup_kn_lock_live+0xe2/0x3e0 [ 1359.045441] #3: ffffffffae568870 (cpu_hotplug_lock){++++}-{0:0}, at: cgroup_procs_write_start+0x3e5/0x650 [ 1359.057087] #4: ffffffffae769a50 (cgroup_threadgroup_rwsem){++++}-{0:0}, at: cgroup_procs_write_start+0x3f1/0x650 [ 1359.069515] [ 1359.071982] =============================================\x0a