[ 785.228103] nipa-hw-worker: drivers/net:napi_threaded.py: timed out [ 785.230578] nipa-hw-worker: [5/44] Retrying drivers/net:napi_threaded.py [ 785.359179] kselftest: Running tests in drivers/net [ 989.712504] INFO: task python3:5837 blocked for more than 122 seconds. [ 989.720708] Not tainted 7.1.0-rc1-fvmh-g646fd76e2503 #1 [ 989.728353] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 989.737946] task:python3 state:D stack:25672 pid:5837 tgid:5837 ppid:5836 task_flags:0x400100 flags:0x00080000 [ 989.751155] Call Trace: [ 989.754695] [ 989.757848] __schedule+0xcba/0x1940 [ 989.762668] ? io_schedule_timeout+0x140/0x140 [ 989.768453] ? find_held_lock+0x2b/0x80 [ 989.773558] ? __lock_release.isra.0+0x6b/0x1a0 [ 989.779445] schedule+0xd4/0x260 [ 989.783862] schedule_preempt_disabled+0x18/0x30 [ 989.789841] __mutex_lock+0x6a3/0x2000 [ 989.794845] ? threaded_store+0x132/0x220 [ 989.800137] ? ww_mutex_lock+0x160/0x160 [ 989.805336] ? selinux_inode_free_security+0x330/0x330 [ 989.811911] ? cap_capable+0x1d6/0x3e0 [ 989.816919] ? threaded_store+0x132/0x220 [ 989.822215] threaded_store+0x132/0x220 [ 989.827319] ? xps_rxqs_store+0x2e0/0x2e0 [ 989.832621] ? sysfs_file_kobj+0xbd/0x1e0 [ 989.837922] ? sysfs_kf_write+0x65/0x160 [ 989.843123] ? sysfs_file_ops+0x140/0x140 [ 989.848418] kernfs_fop_write_iter+0x335/0x510 [ 989.854204] ? kernfs_ops+0xf0/0xf0 [ 989.858915] new_sync_write+0x333/0x750 [ 989.864013] ? __lock_acquire+0x508/0xc10 [ 989.869310] ? new_sync_read+0x740/0x740 [ 989.874512] ? lock_acquire.part.0+0xbc/0x260 [ 989.880200] ? ksys_write+0x116/0x250 [ 989.885113] vfs_write+0x657/0x9b0 [ 989.889729] ksys_write+0x116/0x250 [ 989.894443] ? __ia32_sys_read+0xc0/0xc0 [ 989.899644] ? kernfs_fop_llseek+0x160/0x1f0 [ 989.905234] ? rcu_is_watching+0x15/0xd0 [ 989.910435] do_syscall_64+0xf3/0x650 [ 989.915349] ? trace_hardirqs_off+0xd/0x30 [ 989.920745] ? exc_page_fault+0xda/0xf0 [ 989.925848] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 989.932307] RIP: 0033:0x7f88a7eb522e [ 989.937119] RSP: 002b:00007ffd56461b10 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 989.946419] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f88a7eb522e [ 989.955229] RDX: 0000000000000001 RSI: 0000558316542410 RDI: 0000000000000007 [ 989.964037] RBP: 00007ffd56461b20 R08: 0000000000000000 R09: 0000000000000000 [ 989.972845] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f88a7d2d390 [ 989.981654] R13: 0000558316542410 R14: 0000000000000007 R15: 0000000000000003 [ 989.990466] [ 989.993763] INFO: task python3:5837 is blocked on a mutex likely owned by task python3:3745. [ 990.004034] task:python3 state:D stack:25672 pid:3745 tgid:3745 ppid:3744 task_flags:0x400100 flags:0x00080802 [ 990.017238] Call Trace: [ 990.020775] [ 990.023925] __schedule+0xcba/0x1940 [ 990.028738] ? io_schedule_timeout+0x140/0x140 [ 990.034520] ? find_held_lock+0x2b/0x80 [ 990.039622] ? __try_to_del_timer_sync+0x155/0x1f0 [ 990.045791] schedule+0xd4/0x260 [ 990.050212] schedule_timeout+0x13f/0x290 [ 990.055511] ? hrtimer_nanosleep_restart+0x270/0x270 [ 990.061882] ? trace_contention_end+0xb3/0x1a0 [ 990.067666] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 990.073362] ? ww_mutex_lock+0x160/0x160 [ 990.078563] msleep+0x1b/0x30 [ 990.082688] napi_set_threaded+0x3a5/0x440 [ 990.088083] ? napi_resume_irqs+0x1a0/0x1a0 [ 990.093570] netif_set_threaded+0x257/0x400 [ 990.099064] threaded_store+0x1b8/0x220 [ 990.104168] ? xps_rxqs_store+0x2e0/0x2e0 [ 990.109466] ? sysfs_file_kobj+0xbd/0x1e0 [ 990.114756] ? sysfs_kf_write+0x65/0x160 [ 990.119958] ? sysfs_file_ops+0x140/0x140 [ 990.125254] kernfs_fop_write_iter+0x335/0x510 [ 990.131038] ? kernfs_ops+0xf0/0xf0 [ 990.135744] new_sync_write+0x333/0x750 [ 990.140845] ? __lock_acquire+0x508/0xc10 [ 990.146144] ? new_sync_read+0x740/0x740 [ 990.151349] ? lock_acquire.part.0+0xbc/0x260 [ 990.157036] ? ksys_write+0x116/0x250 [ 990.161949] vfs_write+0x657/0x9b0 [ 990.166565] ksys_write+0x116/0x250 [ 990.171276] ? __ia32_sys_read+0xc0/0xc0 [ 990.176477] ? kernfs_fop_llseek+0x160/0x1f0 [ 990.182067] ? rcu_is_watching+0x15/0xd0 [ 990.187267] do_syscall_64+0xf3/0x650 [ 990.192173] ? irq_exit_rcu+0x1a/0x30 [ 990.197081] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 990.203546] RIP: 0033:0x7fa307f8722e [ 990.208358] RSP: 002b:00007ffc32acb900 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 990.217653] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa307f8722e [ 990.226463] RDX: 0000000000000001 RSI: 000055d01dc24410 RDI: 0000000000000007 [ 990.235271] RBP: 00007ffc32acb910 R08: 0000000000000000 R09: 0000000000000000 [ 990.244082] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa307dff390 [ 990.252890] R13: 000055d01dc24410 R14: 0000000000000007 R15: 0000000000000003 [ 990.261697] [ 990.264957] \x0aShowing all locks held in the system: [ 990.273494] 1 lock held by khungtaskd/185: [ 990.278889] #0: ffffffffb8efdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 990.290379] 4 locks held by python3/3745: [ 990.295672] #0: ff11000131ca13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.305560] #1: ff110001643f2680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.316525] #2: ff11000124650b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.327680] #3: ff11000124631d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.337779] 4 locks held by python3/5837: [ 990.343069] #0: ff11000131ca13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.352958] #1: ff110001ea290280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.363911] #2: ff11000124650b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.375068] #3: ff11000124631d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.385161] [ 990.387627] =============================================\x0a [ 1112.594125] INFO: task python3:5837 blocked for more than 245 seconds. [ 1112.602274] Not tainted 7.1.0-rc1-fvmh-g646fd76e2503 #1 [ 1112.609524] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1112.619113] task:python3 state:D stack:25672 pid:5837 tgid:5837 ppid:5836 task_flags:0x400100 flags:0x00080000 [ 1112.632312] Call Trace: [ 1112.635851] [ 1112.639001] __schedule+0xcba/0x1940 [ 1112.643817] ? io_schedule_timeout+0x140/0x140 [ 1112.649601] ? find_held_lock+0x2b/0x80 [ 1112.654705] ? __lock_release.isra.0+0x6b/0x1a0 [ 1112.660583] schedule+0xd4/0x260 [ 1112.665002] schedule_preempt_disabled+0x18/0x30 [ 1112.670982] __mutex_lock+0x6a3/0x2000 [ 1112.675979] ? threaded_store+0x132/0x220 [ 1112.681275] ? ww_mutex_lock+0x160/0x160 [ 1112.686473] ? selinux_inode_free_security+0x330/0x330 [ 1112.693032] ? cap_capable+0x1d6/0x3e0 [ 1112.698040] ? threaded_store+0x132/0x220 [ 1112.703335] threaded_store+0x132/0x220 [ 1112.708435] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.713732] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.719032] ? sysfs_kf_write+0x65/0x160 [ 1112.724231] ? sysfs_file_ops+0x140/0x140 [ 1112.729528] kernfs_fop_write_iter+0x335/0x510 [ 1112.735310] ? kernfs_ops+0xf0/0xf0 [ 1112.740021] new_sync_write+0x333/0x750 [ 1112.745124] ? __lock_acquire+0x508/0xc10 [ 1112.750420] ? new_sync_read+0x740/0x740 [ 1112.755622] ? lock_acquire.part.0+0xbc/0x260 [ 1112.761305] ? ksys_write+0x116/0x250 [ 1112.766214] vfs_write+0x657/0x9b0 [ 1112.770831] ksys_write+0x116/0x250 [ 1112.775544] ? __ia32_sys_read+0xc0/0xc0 [ 1112.780735] ? kernfs_fop_llseek+0x160/0x1f0 [ 1112.786322] ? rcu_is_watching+0x15/0xd0 [ 1112.791524] do_syscall_64+0xf3/0x650 [ 1112.796430] ? trace_hardirqs_off+0xd/0x30 [ 1112.801826] ? exc_page_fault+0xda/0xf0 [ 1112.806928] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1112.813384] RIP: 0033:0x7f88a7eb522e [ 1112.818195] RSP: 002b:00007ffd56461b10 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1112.827490] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f88a7eb522e [ 1112.836298] RDX: 0000000000000001 RSI: 0000558316542410 RDI: 0000000000000007 [ 1112.845106] RBP: 00007ffd56461b20 R08: 0000000000000000 R09: 0000000000000000 [ 1112.853915] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f88a7d2d390 [ 1112.862723] R13: 0000558316542410 R14: 0000000000000007 R15: 0000000000000003 [ 1112.871537] [ 1112.874820] INFO: task python3:5837 is blocked on a mutex likely owned by task python3:3745. [ 1112.885085] task:python3 state:D stack:25672 pid:3745 tgid:3745 ppid:3744 task_flags:0x400100 flags:0x00080802 [ 1112.898287] Call Trace: [ 1112.901826] [ 1112.904974] __schedule+0xcba/0x1940 [ 1112.909784] ? io_schedule_timeout+0x140/0x140 [ 1112.915560] ? find_held_lock+0x2b/0x80 [ 1112.920661] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1112.926837] schedule+0xd4/0x260 [ 1112.931253] schedule_timeout+0x13f/0x290 [ 1112.936548] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1112.942914] ? trace_contention_end+0xb3/0x1a0 [ 1112.948701] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1112.954388] ? ww_mutex_lock+0x160/0x160 [ 1112.959589] msleep+0x1b/0x30 [ 1112.963712] napi_set_threaded+0x3a5/0x440 [ 1112.969107] ? napi_resume_irqs+0x1a0/0x1a0 [ 1112.974603] netif_set_threaded+0x257/0x400 [ 1112.980097] threaded_store+0x1b8/0x220 [ 1112.985200] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.990489] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.995775] ? sysfs_kf_write+0x65/0x160 [ 1113.000976] ? sysfs_file_ops+0x140/0x140 [ 1113.006272] kernfs_fop_write_iter+0x335/0x510 [ 1113.012055] ? kernfs_ops+0xf0/0xf0 [ 1113.016767] new_sync_write+0x333/0x750 [ 1113.021870] ? __lock_acquire+0x508/0xc10 [ 1113.027156] ? new_sync_read+0x740/0x740 [ 1113.032357] ? lock_acquire.part.0+0xbc/0x260 [ 1113.038040] ? ksys_write+0x116/0x250 [ 1113.042949] vfs_write+0x657/0x9b0 [ 1113.047565] ksys_write+0x116/0x250 [ 1113.052267] ? __ia32_sys_read+0xc0/0xc0 [ 1113.057466] ? kernfs_fop_llseek+0x160/0x1f0 [ 1113.063055] ? rcu_is_watching+0x15/0xd0 [ 1113.068253] do_syscall_64+0xf3/0x650 [ 1113.073160] ? irq_exit_rcu+0x1a/0x30 [ 1113.078056] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1113.084521] RIP: 0033:0x7fa307f8722e [ 1113.089328] RSP: 002b:00007ffc32acb900 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1113.098623] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa307f8722e [ 1113.107429] RDX: 0000000000000001 RSI: 000055d01dc24410 RDI: 0000000000000007 [ 1113.116234] RBP: 00007ffc32acb910 R08: 0000000000000000 R09: 0000000000000000 [ 1113.125041] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa307dff390 [ 1113.133845] R13: 000055d01dc24410 R14: 0000000000000007 R15: 0000000000000003 [ 1113.142655] [ 1113.145918] \x0aShowing all locks held in the system: [ 1113.154457] 1 lock held by khungtaskd/185: [ 1113.159847] #0: ffffffffb8efdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1113.171320] 2 locks held by systemd-journal/809: [ 1113.177298] #0: ff1100180e6b72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940 [ 1113.187184] #1: ffffffffb9103120 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: link_path_walk+0x9a7/0x1830 [ 1113.199810] 4 locks held by python3/3745: [ 1113.205096] #0: ff11000131ca13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.214983] #1: ff110001643f2680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.225944] #2: ff11000124650b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.237102] #3: ff11000124631d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.247199] 4 locks held by python3/5837: [ 1113.252493] #0: ff11000131ca13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.262384] #1: ff110001ea290280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.273343] #2: ff11000124650b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.284498] #3: ff11000124631d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.294589] [ 1113.297052] =============================================\x0a [ 1235.475722] INFO: task python3:5837 blocked for more than 368 seconds. [ 1235.483869] Not tainted 7.1.0-rc1-fvmh-g646fd76e2503 #1 [ 1235.491110] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1235.500699] task:python3 state:D stack:25672 pid:5837 tgid:5837 ppid:5836 task_flags:0x400100 flags:0x00080002 [ 1235.513903] Call Trace: [ 1235.517443] [ 1235.520591] __schedule+0xcba/0x1940 [ 1235.525406] ? io_schedule_timeout+0x140/0x140 [ 1235.531179] ? find_held_lock+0x2b/0x80 [ 1235.536286] ? __lock_release.isra.0+0x6b/0x1a0 [ 1235.542170] schedule+0xd4/0x260 [ 1235.546586] schedule_preempt_disabled+0x18/0x30 [ 1235.552571] __mutex_lock+0x6a3/0x2000 [ 1235.557574] ? threaded_store+0x132/0x220 [ 1235.562873] ? ww_mutex_lock+0x160/0x160 [ 1235.568070] ? selinux_inode_free_security+0x330/0x330 [ 1235.574641] ? cap_capable+0x1d6/0x3e0 [ 1235.579645] ? threaded_store+0x132/0x220 [ 1235.584942] threaded_store+0x132/0x220 [ 1235.590044] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.595343] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.600643] ? sysfs_kf_write+0x65/0x160 [ 1235.605840] ? sysfs_file_ops+0x140/0x140 [ 1235.611136] kernfs_fop_write_iter+0x335/0x510 [ 1235.616920] ? kernfs_ops+0xf0/0xf0 [ 1235.621630] new_sync_write+0x333/0x750 [ 1235.626734] ? __lock_acquire+0x508/0xc10 [ 1235.632028] ? new_sync_read+0x740/0x740 [ 1235.637229] ? lock_acquire.part.0+0xbc/0x260 [ 1235.642914] ? ksys_write+0x116/0x250 [ 1235.647822] vfs_write+0x657/0x9b0 [ 1235.652436] ksys_write+0x116/0x250 [ 1235.657145] ? __ia32_sys_read+0xc0/0xc0 [ 1235.662344] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.667935] ? rcu_is_watching+0x15/0xd0 [ 1235.673133] do_syscall_64+0xf3/0x650 [ 1235.678041] ? trace_hardirqs_off+0xd/0x30 [ 1235.683424] ? exc_page_fault+0xda/0xf0 [ 1235.688526] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.694991] RIP: 0033:0x7f88a7eb522e [ 1235.699798] RSP: 002b:00007ffd56461b10 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.709095] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f88a7eb522e [ 1235.717892] RDX: 0000000000000001 RSI: 0000558316542410 RDI: 0000000000000007 [ 1235.726701] RBP: 00007ffd56461b20 R08: 0000000000000000 R09: 0000000000000000 [ 1235.735510] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f88a7d2d390 [ 1235.744317] R13: 0000558316542410 R14: 0000000000000007 R15: 0000000000000003 [ 1235.753122] [ 1235.756403] INFO: task python3:5837 is blocked on a mutex likely owned by task python3:3745. [ 1235.766677] task:python3 state:D stack:25672 pid:3745 tgid:3745 ppid:3744 task_flags:0x400100 flags:0x00080802 [ 1235.779879] Call Trace: [ 1235.783415] [ 1235.786569] __schedule+0xcba/0x1940 [ 1235.791376] ? io_schedule_timeout+0x140/0x140 [ 1235.797160] ? find_held_lock+0x2b/0x80 [ 1235.802264] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1235.808439] schedule+0xd4/0x260 [ 1235.812857] schedule_timeout+0x13f/0x290 [ 1235.818151] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1235.824519] ? trace_contention_end+0xb3/0x1a0 [ 1235.830302] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1235.835989] ? ww_mutex_lock+0x160/0x160 [ 1235.841184] msleep+0x1b/0x30 [ 1235.845311] napi_set_threaded+0x3a5/0x440 [ 1235.850705] ? napi_resume_irqs+0x1a0/0x1a0 [ 1235.856200] netif_set_threaded+0x257/0x400 [ 1235.861690] threaded_store+0x1b8/0x220 [ 1235.866793] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.872087] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.877384] ? sysfs_kf_write+0x65/0x160 [ 1235.882584] ? sysfs_file_ops+0x140/0x140 [ 1235.887869] kernfs_fop_write_iter+0x335/0x510 [ 1235.893650] ? kernfs_ops+0xf0/0xf0 [ 1235.898363] new_sync_write+0x333/0x750 [ 1235.903454] ? __lock_acquire+0x508/0xc10 [ 1235.908749] ? new_sync_read+0x740/0x740 [ 1235.913947] ? lock_acquire.part.0+0xbc/0x260 [ 1235.919632] ? ksys_write+0x116/0x250 [ 1235.924542] vfs_write+0x657/0x9b0 [ 1235.929157] ksys_write+0x116/0x250 [ 1235.933868] ? __ia32_sys_read+0xc0/0xc0 [ 1235.939067] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.944655] ? rcu_is_watching+0x15/0xd0 [ 1235.949856] do_syscall_64+0xf3/0x650 [ 1235.954763] ? irq_exit_rcu+0x1a/0x30 [ 1235.959671] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.966138] RIP: 0033:0x7fa307f8722e [ 1235.970937] RSP: 002b:00007ffc32acb900 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.980233] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa307f8722e [ 1235.989032] RDX: 0000000000000001 RSI: 000055d01dc24410 RDI: 0000000000000007 [ 1235.997832] RBP: 00007ffc32acb910 R08: 0000000000000000 R09: 0000000000000000 [ 1236.006641] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa307dff390 [ 1236.015449] R13: 000055d01dc24410 R14: 0000000000000007 R15: 0000000000000003 [ 1236.024260] [ 1236.027525] \x0aShowing all locks held in the system: [ 1236.036064] 1 lock held by khungtaskd/185: [ 1236.041456] #0: ffffffffb8efdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1236.052937] 4 locks held by python3/3745: [ 1236.058230] #0: ff11000131ca13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.068117] #1: ff110001643f2680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.079078] #2: ff11000124650b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.090235] #3: ff11000124631d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.100318] 4 locks held by python3/5837: [ 1236.105611] #0: ff11000131ca13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.115498] #1: ff110001ea290280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.126457] #2: ff11000124650b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.137614] #3: ff11000124631d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.147704] [ 1236.150171] =============================================\x0a [ 1358.357320] INFO: task python3:5837 blocked for more than 491 seconds. [ 1358.365467] Not tainted 7.1.0-rc1-fvmh-g646fd76e2503 #1 [ 1358.372716] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1358.382308] task:python3 state:D stack:25672 pid:5837 tgid:5837 ppid:5836 task_flags:0x400100 flags:0x00080002 [ 1358.395515] Call Trace: [ 1358.399055] [ 1358.402205] __schedule+0xcba/0x1940 [ 1358.407023] ? io_schedule_timeout+0x140/0x140 [ 1358.412811] ? find_held_lock+0x2b/0x80 [ 1358.417918] ? __lock_release.isra.0+0x6b/0x1a0 [ 1358.423803] schedule+0xd4/0x260 [ 1358.428221] schedule_preempt_disabled+0x18/0x30 [ 1358.434202] __mutex_lock+0x6a3/0x2000 [ 1358.439207] ? threaded_store+0x132/0x220 [ 1358.444507] ? ww_mutex_lock+0x160/0x160 [ 1358.449699] ? selinux_inode_free_security+0x330/0x330 [ 1358.456270] ? cap_capable+0x1d6/0x3e0 [ 1358.461278] ? threaded_store+0x132/0x220 [ 1358.466576] threaded_store+0x132/0x220 [ 1358.471679] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.476971] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.482272] ? sysfs_kf_write+0x65/0x160 [ 1358.487471] ? sysfs_file_ops+0x140/0x140 [ 1358.492768] kernfs_fop_write_iter+0x335/0x510 [ 1358.498552] ? kernfs_ops+0xf0/0xf0 [ 1358.503265] new_sync_write+0x333/0x750 [ 1358.508370] ? __lock_acquire+0x508/0xc10 [ 1358.513666] ? new_sync_read+0x740/0x740 [ 1358.518868] ? lock_acquire.part.0+0xbc/0x260 [ 1358.524555] ? ksys_write+0x116/0x250 [ 1358.529456] vfs_write+0x657/0x9b0 [ 1358.534070] ksys_write+0x116/0x250 [ 1358.538784] ? __ia32_sys_read+0xc0/0xc0 [ 1358.543982] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.549571] ? rcu_is_watching+0x15/0xd0 [ 1358.554764] do_syscall_64+0xf3/0x650 [ 1358.559670] ? trace_hardirqs_off+0xd/0x30 [ 1358.565067] ? exc_page_fault+0xda/0xf0 [ 1358.570173] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.576643] RIP: 0033:0x7f88a7eb522e [ 1358.581454] RSP: 002b:00007ffd56461b10 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.590742] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f88a7eb522e [ 1358.599550] RDX: 0000000000000001 RSI: 0000558316542410 RDI: 0000000000000007 [ 1358.608350] RBP: 00007ffd56461b20 R08: 0000000000000000 R09: 0000000000000000 [ 1358.617163] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f88a7d2d390 [ 1358.625975] R13: 0000558316542410 R14: 0000000000000007 R15: 0000000000000003 [ 1358.634788] [ 1358.638060] INFO: task python3:5837 is blocked on a mutex likely owned by task python3:3745. [ 1358.648333] task:python3 state:D stack:25672 pid:3745 tgid:3745 ppid:3744 task_flags:0x400100 flags:0x00080802 [ 1358.661535] Call Trace: [ 1358.665075] [ 1358.668224] __schedule+0xcba/0x1940 [ 1358.673033] ? io_schedule_timeout+0x140/0x140 [ 1358.678821] ? find_held_lock+0x2b/0x80 [ 1358.683925] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1358.690105] schedule+0xd4/0x260 [ 1358.694523] schedule_timeout+0x13f/0x290 [ 1358.699821] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1358.706184] ? trace_contention_end+0xb3/0x1a0 [ 1358.711971] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1358.717661] ? ww_mutex_lock+0x160/0x160 [ 1358.722862] msleep+0x1b/0x30 [ 1358.726988] napi_set_threaded+0x3a5/0x440 [ 1358.732382] ? napi_resume_irqs+0x1a0/0x1a0 [ 1358.737875] netif_set_threaded+0x257/0x400 [ 1358.743362] threaded_store+0x1b8/0x220 [ 1358.748463] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.753763] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.759061] ? sysfs_kf_write+0x65/0x160 [ 1358.764262] ? sysfs_file_ops+0x140/0x140 [ 1358.769559] kernfs_fop_write_iter+0x335/0x510 [ 1358.775344] ? kernfs_ops+0xf0/0xf0 [ 1358.780056] new_sync_write+0x333/0x750 [ 1358.785168] ? __lock_acquire+0x508/0xc10 [ 1358.790462] ? new_sync_read+0x740/0x740 [ 1358.795662] ? lock_acquire.part.0+0xbc/0x260 [ 1358.801348] ? ksys_write+0x116/0x250 [ 1358.806257] vfs_write+0x657/0x9b0 [ 1358.810872] ksys_write+0x116/0x250 [ 1358.815585] ? __ia32_sys_read+0xc0/0xc0 [ 1358.820788] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.826378] ? rcu_is_watching+0x15/0xd0 [ 1358.831579] do_syscall_64+0xf3/0x650 [ 1358.836488] ? irq_exit_rcu+0x1a/0x30 [ 1358.841399] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.847864] RIP: 0033:0x7fa307f8722e [ 1358.852672] RSP: 002b:00007ffc32acb900 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.861969] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa307f8722e [ 1358.870778] RDX: 0000000000000001 RSI: 000055d01dc24410 RDI: 0000000000000007 [ 1358.879584] RBP: 00007ffc32acb910 R08: 0000000000000000 R09: 0000000000000000 [ 1358.888392] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa307dff390 [ 1358.897201] R13: 000055d01dc24410 R14: 0000000000000007 R15: 0000000000000003 [ 1358.906013] [ 1358.909278] \x0aShowing all locks held in the system: [ 1358.917816] 1 lock held by khungtaskd/185: [ 1358.923211] #0: ffffffffb8efdee0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1358.934685] 4 locks held by python3/3745: [ 1358.939982] #0: ff11000131ca13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1358.949870] #1: ff110001643f2680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1358.960835] #2: ff11000124650b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1358.971993] #3: ff11000124631d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1358.982088] 4 locks held by python3/5837: [ 1358.987384] #0: ff11000131ca13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1358.997273] #1: ff110001ea290280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1359.008228] #2: ff11000124650b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1359.019384] #3: ff11000124631d60 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1359.029479] [ 1359.031946] =============================================\x0a