[ 793.180574] nipa-hw-worker: drivers/net:napi_threaded.py: timed out [ 793.182988] nipa-hw-worker: [5/44] Retrying drivers/net:napi_threaded.py [ 793.336267] kselftest: Running tests in drivers/net [ 969.213926] kworker/u130:6 (852) used greatest stack depth: 21456 bytes left [ 989.693189] INFO: task python3:5948 blocked for more than 122 seconds. [ 989.701380] Not tainted 7.1.0-rc1-kcuo-ga15578502849 #1 [ 989.708842] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 989.718433] task:python3 state:D stack:25032 pid:5948 tgid:5948 ppid:5947 task_flags:0x400100 flags:0x00080000 [ 989.731638] Call Trace: [ 989.735175] [ 989.738324] __schedule+0xcba/0x1940 [ 989.743141] ? io_schedule_timeout+0x140/0x140 [ 989.748924] ? find_held_lock+0x2b/0x80 [ 989.754028] ? __lock_release.isra.0+0x6b/0x1a0 [ 989.759912] schedule+0xd4/0x260 [ 989.764328] schedule_preempt_disabled+0x18/0x30 [ 989.770306] __mutex_lock+0x6a3/0x2000 [ 989.775311] ? threaded_store+0x132/0x220 [ 989.780613] ? ww_mutex_lock+0x160/0x160 [ 989.785810] ? selinux_inode_free_security+0x330/0x330 [ 989.792380] ? cap_capable+0x1d6/0x3e0 [ 989.797386] ? threaded_store+0x132/0x220 [ 989.802681] threaded_store+0x132/0x220 [ 989.807784] ? xps_rxqs_store+0x2e0/0x2e0 [ 989.813082] ? sysfs_file_kobj+0xbd/0x1e0 [ 989.818380] ? sysfs_kf_write+0x65/0x160 [ 989.823578] ? sysfs_file_ops+0x140/0x140 [ 989.828872] kernfs_fop_write_iter+0x335/0x510 [ 989.834655] ? kernfs_ops+0xf0/0xf0 [ 989.839366] new_sync_write+0x333/0x750 [ 989.844469] ? __lock_acquire+0x508/0xc10 [ 989.849767] ? new_sync_read+0x740/0x740 [ 989.854966] ? lock_acquire.part.0+0xbc/0x260 [ 989.860650] ? ksys_write+0x116/0x250 [ 989.865559] vfs_write+0x657/0x9b0 [ 989.870172] ksys_write+0x116/0x250 [ 989.874882] ? __ia32_sys_read+0xc0/0xc0 [ 989.880082] ? kernfs_fop_llseek+0x160/0x1f0 [ 989.885671] ? rcu_is_watching+0x15/0xd0 [ 989.890874] do_syscall_64+0xf3/0x650 [ 989.895779] ? trace_hardirqs_off+0xd/0x30 [ 989.901172] ? exc_page_fault+0xda/0xf0 [ 989.906271] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 989.912739] RIP: 0033:0x7f6197d7122e [ 989.917547] RSP: 002b:00007ffcac5389f0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 989.926845] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f6197d7122e [ 989.935651] RDX: 0000000000000001 RSI: 000055a0b2e4da10 RDI: 0000000000000007 [ 989.944459] RBP: 00007ffcac538a00 R08: 0000000000000000 R09: 0000000000000000 [ 989.953258] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f6197be9390 [ 989.962066] R13: 000055a0b2e4da10 R14: 0000000000000007 R15: 0000000000000003 [ 989.970871] [ 989.974175] INFO: task python3:5948 is blocked on a mutex likely owned by task python3:3915. [ 989.984446] task:python3 state:D stack:25472 pid:3915 tgid:3915 ppid:3914 task_flags:0x400100 flags:0x00080802 [ 989.997647] Call Trace: [ 990.001185] [ 990.004336] __schedule+0xcba/0x1940 [ 990.009136] ? io_schedule_timeout+0x140/0x140 [ 990.014919] ? find_held_lock+0x2b/0x80 [ 990.020022] ? __try_to_del_timer_sync+0x155/0x1f0 [ 990.026198] schedule+0xd4/0x260 [ 990.030615] schedule_timeout+0x13f/0x290 [ 990.035910] ? hrtimer_nanosleep_restart+0x270/0x270 [ 990.042278] ? trace_contention_end+0xb3/0x1a0 [ 990.048063] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 990.053762] ? ww_mutex_lock+0x160/0x160 [ 990.058964] msleep+0x1b/0x30 [ 990.063086] napi_set_threaded+0x3a5/0x440 [ 990.068482] ? napi_resume_irqs+0x1a0/0x1a0 [ 990.073967] netif_set_threaded+0x257/0x400 [ 990.079459] threaded_store+0x1b8/0x220 [ 990.084559] ? xps_rxqs_store+0x2e0/0x2e0 [ 990.089855] ? sysfs_file_kobj+0xbd/0x1e0 [ 990.095151] ? sysfs_kf_write+0x65/0x160 [ 990.100349] ? sysfs_file_ops+0x140/0x140 [ 990.105645] kernfs_fop_write_iter+0x335/0x510 [ 990.111431] ? kernfs_ops+0xf0/0xf0 [ 990.116139] new_sync_write+0x333/0x750 [ 990.121240] ? __lock_acquire+0x508/0xc10 [ 990.126533] ? new_sync_read+0x740/0x740 [ 990.131732] ? lock_acquire.part.0+0xbc/0x260 [ 990.137416] ? ksys_write+0x116/0x250 [ 990.142323] vfs_write+0x657/0x9b0 [ 990.146937] ksys_write+0x116/0x250 [ 990.151649] ? __ia32_sys_read+0xc0/0xc0 [ 990.156845] ? kernfs_fop_llseek+0x160/0x1f0 [ 990.162433] ? rcu_is_watching+0x15/0xd0 [ 990.167631] do_syscall_64+0xf3/0x650 [ 990.172533] ? irq_exit_rcu+0x1a/0x30 [ 990.177439] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 990.183906] RIP: 0033:0x7fa88734c22e [ 990.188711] RSP: 002b:00007ffcd8027df0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 990.198005] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa88734c22e [ 990.206810] RDX: 0000000000000001 RSI: 000055b645d4ea10 RDI: 0000000000000007 [ 990.215618] RBP: 00007ffcd8027e00 R08: 0000000000000000 R09: 0000000000000000 [ 990.224425] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa8871c4390 [ 990.233233] R13: 000055b645d4ea10 R14: 0000000000000007 R15: 0000000000000003 [ 990.242043] [ 990.245301] \x0aShowing all locks held in the system: [ 990.253842] 1 lock held by khungtaskd/185: [ 990.259235] #0: ffffffffa6cfdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 990.270719] 4 locks held by python3/3915: [ 990.276018] #0: ff110017dab243e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.285898] #1: ff11000163183280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.296851] #2: ff11000122ea4b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.308004] #3: ff11000122f09d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.318262] 4 locks held by python3/5948: [ 990.323559] #0: ff110017dab243e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.333446] #1: ff110001f1b15080 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.344405] #2: ff11000122ea4b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.355563] #3: ff11000122f09d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.365655] [ 990.368120] =============================================\x0a [ 1112.574361] INFO: task python3:5948 blocked for more than 245 seconds. [ 1112.582510] Not tainted 7.1.0-rc1-kcuo-ga15578502849 #1 [ 1112.589761] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1112.599350] task:python3 state:D stack:25032 pid:5948 tgid:5948 ppid:5947 task_flags:0x400100 flags:0x00080000 [ 1112.612554] Call Trace: [ 1112.616096] [ 1112.619250] __schedule+0xcba/0x1940 [ 1112.624068] ? io_schedule_timeout+0x140/0x140 [ 1112.629854] ? find_held_lock+0x2b/0x80 [ 1112.634961] ? __lock_release.isra.0+0x6b/0x1a0 [ 1112.640847] schedule+0xd4/0x260 [ 1112.645264] schedule_preempt_disabled+0x18/0x30 [ 1112.651246] __mutex_lock+0x6a3/0x2000 [ 1112.656251] ? threaded_store+0x132/0x220 [ 1112.661550] ? ww_mutex_lock+0x160/0x160 [ 1112.666741] ? selinux_inode_free_security+0x330/0x330 [ 1112.673312] ? cap_capable+0x1d6/0x3e0 [ 1112.678328] ? threaded_store+0x132/0x220 [ 1112.683634] threaded_store+0x132/0x220 [ 1112.688733] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.694029] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.699322] ? sysfs_kf_write+0x65/0x160 [ 1112.704524] ? sysfs_file_ops+0x140/0x140 [ 1112.709820] kernfs_fop_write_iter+0x335/0x510 [ 1112.715607] ? kernfs_ops+0xf0/0xf0 [ 1112.720318] new_sync_write+0x333/0x750 [ 1112.725424] ? __lock_acquire+0x508/0xc10 [ 1112.730720] ? new_sync_read+0x740/0x740 [ 1112.735922] ? lock_acquire.part.0+0xbc/0x260 [ 1112.741609] ? ksys_write+0x116/0x250 [ 1112.746512] vfs_write+0x657/0x9b0 [ 1112.751129] ksys_write+0x116/0x250 [ 1112.755842] ? __ia32_sys_read+0xc0/0xc0 [ 1112.761041] ? kernfs_fop_llseek+0x160/0x1f0 [ 1112.766632] ? rcu_is_watching+0x15/0xd0 [ 1112.771834] do_syscall_64+0xf3/0x650 [ 1112.776742] ? trace_hardirqs_off+0xd/0x30 [ 1112.782138] ? exc_page_fault+0xda/0xf0 [ 1112.787242] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1112.793712] RIP: 0033:0x7f6197d7122e [ 1112.798521] RSP: 002b:00007ffcac5389f0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1112.807820] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f6197d7122e [ 1112.816629] RDX: 0000000000000001 RSI: 000055a0b2e4da10 RDI: 0000000000000007 [ 1112.825439] RBP: 00007ffcac538a00 R08: 0000000000000000 R09: 0000000000000000 [ 1112.834251] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f6197be9390 [ 1112.843058] R13: 000055a0b2e4da10 R14: 0000000000000007 R15: 0000000000000003 [ 1112.851872] [ 1112.855150] INFO: task python3:5948 is blocked on a mutex likely owned by task python3:3915. [ 1112.865425] task:python3 state:D stack:25472 pid:3915 tgid:3915 ppid:3914 task_flags:0x400100 flags:0x00080802 [ 1112.878628] Call Trace: [ 1112.882167] [ 1112.885317] __schedule+0xcba/0x1940 [ 1112.890129] ? io_schedule_timeout+0x140/0x140 [ 1112.895915] ? find_held_lock+0x2b/0x80 [ 1112.901018] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1112.907198] schedule+0xd4/0x260 [ 1112.911616] schedule_timeout+0x13f/0x290 [ 1112.916912] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1112.923282] ? trace_contention_end+0xb3/0x1a0 [ 1112.929060] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1112.934751] ? ww_mutex_lock+0x160/0x160 [ 1112.939951] msleep+0x1b/0x30 [ 1112.944076] napi_set_threaded+0x3a5/0x440 [ 1112.949472] ? napi_resume_irqs+0x1a0/0x1a0 [ 1112.954966] netif_set_threaded+0x257/0x400 [ 1112.960459] threaded_store+0x1b8/0x220 [ 1112.965563] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.970860] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.976158] ? sysfs_kf_write+0x65/0x160 [ 1112.981349] ? sysfs_file_ops+0x140/0x140 [ 1112.986637] kernfs_fop_write_iter+0x335/0x510 [ 1112.992423] ? kernfs_ops+0xf0/0xf0 [ 1112.997133] new_sync_write+0x333/0x750 [ 1113.002235] ? __lock_acquire+0x508/0xc10 [ 1113.007531] ? new_sync_read+0x740/0x740 [ 1113.012731] ? lock_acquire.part.0+0xbc/0x260 [ 1113.018416] ? ksys_write+0x116/0x250 [ 1113.023329] vfs_write+0x657/0x9b0 [ 1113.027945] ksys_write+0x116/0x250 [ 1113.032659] ? __ia32_sys_read+0xc0/0xc0 [ 1113.037859] ? kernfs_fop_llseek+0x160/0x1f0 [ 1113.043450] ? rcu_is_watching+0x15/0xd0 [ 1113.048651] do_syscall_64+0xf3/0x650 [ 1113.053561] ? irq_exit_rcu+0x1a/0x30 [ 1113.058469] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1113.064936] RIP: 0033:0x7fa88734c22e [ 1113.069744] RSP: 002b:00007ffcd8027df0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1113.079040] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa88734c22e [ 1113.087839] RDX: 0000000000000001 RSI: 000055b645d4ea10 RDI: 0000000000000007 [ 1113.096641] RBP: 00007ffcd8027e00 R08: 0000000000000000 R09: 0000000000000000 [ 1113.105451] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa8871c4390 [ 1113.114261] R13: 000055b645d4ea10 R14: 0000000000000007 R15: 0000000000000003 [ 1113.123074] [ 1113.126335] \x0aShowing all locks held in the system: [ 1113.134874] 1 lock held by khungtaskd/185: [ 1113.140269] #0: ffffffffa6cfdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1113.151743] 2 locks held by systemd-journal/804: [ 1113.157713] #0: ff1100180e3b72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940 [ 1113.167602] #1: ff110001a2d8ef78 (&mm->mmap_lock){++++}-{4:4}, at: fs_reclaim_acquire+0x7b/0x120 [ 1113.178382] 4 locks held by python3/3915: [ 1113.183675] #0: ff110017dab243e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.193567] #1: ff11000163183280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.204529] #2: ff11000122ea4b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.215688] #3: ff11000122f09d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.225783] 4 locks held by python3/5948: [ 1113.231079] #0: ff110017dab243e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.240967] #1: ff110001f1b15080 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.251930] #2: ff11000122ea4b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.263085] #3: ff11000122f09d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.273173] [ 1113.275641] =============================================\x0a [ 1235.456524] INFO: task python3:5948 blocked for more than 368 seconds. [ 1235.464666] Not tainted 7.1.0-rc1-kcuo-ga15578502849 #1 [ 1235.471914] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1235.481500] task:python3 state:D stack:25032 pid:5948 tgid:5948 ppid:5947 task_flags:0x400100 flags:0x00080002 [ 1235.494704] Call Trace: [ 1235.498243] [ 1235.501393] __schedule+0xcba/0x1940 [ 1235.506210] ? io_schedule_timeout+0x140/0x140 [ 1235.511996] ? find_held_lock+0x2b/0x80 [ 1235.517104] ? __lock_release.isra.0+0x6b/0x1a0 [ 1235.522988] schedule+0xd4/0x260 [ 1235.527408] schedule_preempt_disabled+0x18/0x30 [ 1235.533387] __mutex_lock+0x6a3/0x2000 [ 1235.538380] ? threaded_store+0x132/0x220 [ 1235.543678] ? ww_mutex_lock+0x160/0x160 [ 1235.548876] ? selinux_inode_free_security+0x330/0x330 [ 1235.555445] ? cap_capable+0x1d6/0x3e0 [ 1235.560454] ? threaded_store+0x132/0x220 [ 1235.565753] threaded_store+0x132/0x220 [ 1235.570853] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.576148] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.581444] ? sysfs_kf_write+0x65/0x160 [ 1235.586643] ? sysfs_file_ops+0x140/0x140 [ 1235.591938] kernfs_fop_write_iter+0x335/0x510 [ 1235.597720] ? kernfs_ops+0xf0/0xf0 [ 1235.602431] new_sync_write+0x333/0x750 [ 1235.607526] ? __lock_acquire+0x508/0xc10 [ 1235.612820] ? new_sync_read+0x740/0x740 [ 1235.618021] ? lock_acquire.part.0+0xbc/0x260 [ 1235.623706] ? ksys_write+0x116/0x250 [ 1235.628613] vfs_write+0x657/0x9b0 [ 1235.633228] ksys_write+0x116/0x250 [ 1235.637932] ? __ia32_sys_read+0xc0/0xc0 [ 1235.643131] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.648718] ? rcu_is_watching+0x15/0xd0 [ 1235.653918] do_syscall_64+0xf3/0x650 [ 1235.658817] ? trace_hardirqs_off+0xd/0x30 [ 1235.664213] ? exc_page_fault+0xda/0xf0 [ 1235.669314] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.675780] RIP: 0033:0x7f6197d7122e [ 1235.680589] RSP: 002b:00007ffcac5389f0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.689879] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f6197d7122e [ 1235.698685] RDX: 0000000000000001 RSI: 000055a0b2e4da10 RDI: 0000000000000007 [ 1235.707491] RBP: 00007ffcac538a00 R08: 0000000000000000 R09: 0000000000000000 [ 1235.716297] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f6197be9390 [ 1235.725102] R13: 000055a0b2e4da10 R14: 0000000000000007 R15: 0000000000000003 [ 1235.733916] [ 1235.737203] INFO: task python3:5948 is blocked on a mutex likely owned by task python3:3915. [ 1235.747475] task:python3 state:D stack:25472 pid:3915 tgid:3915 ppid:3914 task_flags:0x400100 flags:0x00080802 [ 1235.760676] Call Trace: [ 1235.764215] [ 1235.767363] __schedule+0xcba/0x1940 [ 1235.772175] ? io_schedule_timeout+0x140/0x140 [ 1235.777961] ? find_held_lock+0x2b/0x80 [ 1235.783064] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1235.789240] schedule+0xd4/0x260 [ 1235.793659] schedule_timeout+0x13f/0x290 [ 1235.798954] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1235.805324] ? trace_contention_end+0xb3/0x1a0 [ 1235.811107] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1235.816794] ? ww_mutex_lock+0x160/0x160 [ 1235.821991] msleep+0x1b/0x30 [ 1235.826113] napi_set_threaded+0x3a5/0x440 [ 1235.831507] ? napi_resume_irqs+0x1a0/0x1a0 [ 1235.837001] netif_set_threaded+0x257/0x400 [ 1235.842492] threaded_store+0x1b8/0x220 [ 1235.847592] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.852890] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.858175] ? sysfs_kf_write+0x65/0x160 [ 1235.863374] ? sysfs_file_ops+0x140/0x140 [ 1235.868668] kernfs_fop_write_iter+0x335/0x510 [ 1235.874451] ? kernfs_ops+0xf0/0xf0 [ 1235.879162] new_sync_write+0x333/0x750 [ 1235.884264] ? __lock_acquire+0x508/0xc10 [ 1235.889558] ? new_sync_read+0x740/0x740 [ 1235.894756] ? lock_acquire.part.0+0xbc/0x260 [ 1235.900440] ? ksys_write+0x116/0x250 [ 1235.905355] vfs_write+0x657/0x9b0 [ 1235.909969] ksys_write+0x116/0x250 [ 1235.914682] ? __ia32_sys_read+0xc0/0xc0 [ 1235.919880] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.925466] ? rcu_is_watching+0x15/0xd0 [ 1235.930665] do_syscall_64+0xf3/0x650 [ 1235.935571] ? irq_exit_rcu+0x1a/0x30 [ 1235.940467] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.946933] RIP: 0033:0x7fa88734c22e [ 1235.951738] RSP: 002b:00007ffcd8027df0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.961035] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa88734c22e [ 1235.969841] RDX: 0000000000000001 RSI: 000055b645d4ea10 RDI: 0000000000000007 [ 1235.978649] RBP: 00007ffcd8027e00 R08: 0000000000000000 R09: 0000000000000000 [ 1235.987455] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa8871c4390 [ 1235.996263] R13: 000055b645d4ea10 R14: 0000000000000007 R15: 0000000000000003 [ 1236.005075] [ 1236.008340] \x0aShowing all locks held in the system: [ 1236.016869] 1 lock held by khungtaskd/185: [ 1236.022263] #0: ffffffffa6cfdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1236.033743] 4 locks held by python3/3915: [ 1236.039037] #0: ff110017dab243e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.048927] #1: ff11000163183280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.059879] #2: ff11000122ea4b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.071033] #3: ff11000122f09d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.081126] 4 locks held by python3/5948: [ 1236.086418] #0: ff110017dab243e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.096305] #1: ff110001f1b15080 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.107265] #2: ff11000122ea4b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.118420] #3: ff11000122f09d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.128510] [ 1236.130975] =============================================\x0a [ 1358.337672] INFO: task python3:5948 blocked for more than 491 seconds. [ 1358.345819] Not tainted 7.1.0-rc1-kcuo-ga15578502849 #1 [ 1358.353068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1358.362659] task:python3 state:D stack:25032 pid:5948 tgid:5948 ppid:5947 task_flags:0x400100 flags:0x00080002 [ 1358.375864] Call Trace: [ 1358.379402] [ 1358.382554] __schedule+0xcba/0x1940 [ 1358.387371] ? io_schedule_timeout+0x140/0x140 [ 1358.393156] ? find_held_lock+0x2b/0x80 [ 1358.398258] ? __lock_release.isra.0+0x6b/0x1a0 [ 1358.404142] schedule+0xd4/0x260 [ 1358.408562] schedule_preempt_disabled+0x18/0x30 [ 1358.414538] __mutex_lock+0x6a3/0x2000 [ 1358.419541] ? threaded_store+0x132/0x220 [ 1358.424842] ? ww_mutex_lock+0x160/0x160 [ 1358.430040] ? selinux_inode_free_security+0x330/0x330 [ 1358.436610] ? cap_capable+0x1d6/0x3e0 [ 1358.441617] ? threaded_store+0x132/0x220 [ 1358.446914] threaded_store+0x132/0x220 [ 1358.452017] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.457304] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.462604] ? sysfs_kf_write+0x65/0x160 [ 1358.467807] ? sysfs_file_ops+0x140/0x140 [ 1358.473102] kernfs_fop_write_iter+0x335/0x510 [ 1358.478887] ? kernfs_ops+0xf0/0xf0 [ 1358.483588] new_sync_write+0x333/0x750 [ 1358.488693] ? __lock_acquire+0x508/0xc10 [ 1358.493977] ? new_sync_read+0x740/0x740 [ 1358.499179] ? lock_acquire.part.0+0xbc/0x260 [ 1358.504865] ? ksys_write+0x116/0x250 [ 1358.509773] vfs_write+0x657/0x9b0 [ 1358.514381] ksys_write+0x116/0x250 [ 1358.519093] ? __ia32_sys_read+0xc0/0xc0 [ 1358.524281] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.529871] ? rcu_is_watching+0x15/0xd0 [ 1358.535073] do_syscall_64+0xf3/0x650 [ 1358.539981] ? trace_hardirqs_off+0xd/0x30 [ 1358.545376] ? exc_page_fault+0xda/0xf0 [ 1358.550478] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.556944] RIP: 0033:0x7f6197d7122e [ 1358.561754] RSP: 002b:00007ffcac5389f0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.571052] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f6197d7122e [ 1358.579858] RDX: 0000000000000001 RSI: 000055a0b2e4da10 RDI: 0000000000000007 [ 1358.588664] RBP: 00007ffcac538a00 R08: 0000000000000000 R09: 0000000000000000 [ 1358.597469] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f6197be9390 [ 1358.606276] R13: 000055a0b2e4da10 R14: 0000000000000007 R15: 0000000000000003 [ 1358.615090] [ 1358.618363] INFO: task python3:5948 is blocked on a mutex likely owned by task python3:3915. [ 1358.628636] task:python3 state:D stack:25472 pid:3915 tgid:3915 ppid:3914 task_flags:0x400100 flags:0x00080802 [ 1358.641838] Call Trace: [ 1358.645378] [ 1358.648525] __schedule+0xcba/0x1940 [ 1358.653333] ? io_schedule_timeout+0x140/0x140 [ 1358.659117] ? find_held_lock+0x2b/0x80 [ 1358.664220] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1358.670397] schedule+0xd4/0x260 [ 1358.674816] schedule_timeout+0x13f/0x290 [ 1358.680112] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1358.686482] ? trace_contention_end+0xb3/0x1a0 [ 1358.692264] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1358.697953] ? ww_mutex_lock+0x160/0x160 [ 1358.703144] msleep+0x1b/0x30 [ 1358.707269] napi_set_threaded+0x3a5/0x440 [ 1358.712662] ? napi_resume_irqs+0x1a0/0x1a0 [ 1358.718156] netif_set_threaded+0x257/0x400 [ 1358.723647] threaded_store+0x1b8/0x220 [ 1358.728748] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.734047] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.739341] ? sysfs_kf_write+0x65/0x160 [ 1358.744541] ? sysfs_file_ops+0x140/0x140 [ 1358.749828] kernfs_fop_write_iter+0x335/0x510 [ 1358.755611] ? kernfs_ops+0xf0/0xf0 [ 1358.760324] new_sync_write+0x333/0x750 [ 1358.765428] ? __lock_acquire+0x508/0xc10 [ 1358.770722] ? new_sync_read+0x740/0x740 [ 1358.775920] ? lock_acquire.part.0+0xbc/0x260 [ 1358.781603] ? ksys_write+0x116/0x250 [ 1358.786519] vfs_write+0x657/0x9b0 [ 1358.791126] ksys_write+0x116/0x250 [ 1358.795836] ? __ia32_sys_read+0xc0/0xc0 [ 1358.801034] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.806622] ? rcu_is_watching+0x15/0xd0 [ 1358.811822] do_syscall_64+0xf3/0x650 [ 1358.816729] ? irq_exit_rcu+0x1a/0x30 [ 1358.821637] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.828102] RIP: 0033:0x7fa88734c22e [ 1358.832911] RSP: 002b:00007ffcd8027df0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.842205] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa88734c22e [ 1358.851012] RDX: 0000000000000001 RSI: 000055b645d4ea10 RDI: 0000000000000007 [ 1358.859816] RBP: 00007ffcd8027e00 R08: 0000000000000000 R09: 0000000000000000 [ 1358.868624] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa8871c4390 [ 1358.877430] R13: 000055b645d4ea10 R14: 0000000000000007 R15: 0000000000000003 [ 1358.886240] [ 1358.889515] \x0aShowing all locks held in the system: [ 1358.898049] 1 lock held by khungtaskd/185: [ 1358.903442] #0: ffffffffa6cfdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1358.914922] 4 locks held by python3/3915: [ 1358.920216] #0: ff110017dab243e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1358.930102] #1: ff11000163183280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1358.941065] #2: ff11000122ea4b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1358.952221] #3: ff11000122f09d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1358.962311] 4 locks held by python3/5948: [ 1358.967604] #0: ff110017dab243e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1358.977493] #1: ff110001f1b15080 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1358.988443] #2: ff11000122ea4b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1358.999598] #3: ff11000122f09d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1359.009692] [ 1359.012157] =============================================\x0a