[ 788.797452] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 788.799837] nipa-hw-worker: [5/46] Retrying drivers/net:napi_threaded.py
[ 788.949099] kselftest: Running tests in drivers/net
[ 989.694037] INFO: task python3:5809 blocked for more than 122 seconds.
[ 989.702262] Not tainted 7.1.0-rc4-rhie-g7ec97933f2d1 #1
[ 989.710127] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.719715] task:python3 state:D stack:25536 pid:5809 tgid:5809 ppid:5808 task_flags:0x400100 flags:0x00080000
[ 989.732921] Call Trace:
[ 989.736458]
[ 989.739608] __schedule+0xcc3/0x19c0
[ 989.744425] ? io_schedule_timeout+0x140/0x140
[ 989.750205] ? find_held_lock+0x2b/0x80
[ 989.755311] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.761192] schedule+0xd8/0x260
[ 989.765610] schedule_preempt_disabled+0x18/0x30
[ 989.771588] __mutex_lock+0x6af/0x2010
[ 989.776592] ? threaded_store+0x132/0x220
[ 989.781892] ? ww_mutex_lock+0x160/0x160
[ 989.787080] ? selinux_inode_free_security+0x330/0x330
[ 989.793647] ? cap_capable+0x1d6/0x3e0
[ 989.798652] ? threaded_store+0x132/0x220
[ 989.803950] threaded_store+0x132/0x220
[ 989.809043] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.814342] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.819638] ? sysfs_kf_write+0x65/0x160
[ 989.824837] ? sysfs_file_ops+0x140/0x140
[ 989.830133] kernfs_fop_write_iter+0x335/0x510
[ 989.835914] ? kernfs_ops+0xf0/0xf0
[ 989.840625] new_sync_write+0x333/0x750
[ 989.845726] ? __lock_acquire+0x508/0xc10
[ 989.851020] ? new_sync_read+0x740/0x740
[ 989.856220] ? lock_acquire.part.0+0xbc/0x260
[ 989.861897] ? ksys_write+0x116/0x250
[ 989.866806] vfs_write+0x657/0x9b0
[ 989.871418] ksys_write+0x116/0x250
[ 989.876127] ? __ia32_sys_read+0xc0/0xc0
[ 989.881323] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.886909] ? rcu_is_watching+0x15/0xd0
[ 989.892109] do_syscall_64+0xf3/0x650
[ 989.897012] ? trace_hardirqs_off+0xd/0x30
[ 989.902409] ? exc_page_fault+0xda/0xf0
[ 989.907501] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.913967] RIP: 0033:0x7fc67b05d22e
[ 989.918777] RSP: 002b:00007ffd39a890a0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.928064] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc67b05d22e
[ 989.936874] RDX: 0000000000000001 RSI: 0000562e8695f6b0 RDI: 0000000000000007
[ 989.945683] RBP: 00007ffd39a890b0 R08: 0000000000000000 R09: 0000000000000000
[ 989.954489] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc67aed5390
[ 989.963297] R13: 0000562e8695f6b0 R14: 0000000000000007 R15: 0000000000000003
[ 989.972120]
[ 989.975417] INFO: task python3:5809 is blocked on a mutex likely owned by task python3:3770.
[ 989.985690] task:python3 state:D stack:25672 pid:3770 tgid:3770 ppid:3769 task_flags:0x400100 flags:0x00080802
[ 989.998892] Call Trace:
[ 990.002430]
[ 990.005580] __schedule+0xcc3/0x19c0
[ 990.010391] ? io_schedule_timeout+0x140/0x140
[ 990.016175] ? find_held_lock+0x2b/0x80
[ 990.021279] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.027457] schedule+0xd8/0x260
[ 990.031875] schedule_timeout+0x13f/0x290
[ 990.037172] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.043542] ? sysvec_apic_timer_interrupt+0xaa/0xe0
[ 990.049904] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.055591] ? __free_object+0x2ad/0x490
[ 990.060791] msleep+0x1b/0x30
[ 990.064916] napi_set_threaded+0x3a5/0x440
[ 990.070309] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.075802] netif_set_threaded+0x257/0x400
[ 990.081288] threaded_store+0x1b8/0x220
[ 990.086389] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.091686] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.096983] ? sysfs_file_ops+0x140/0x140
[ 990.102267] kernfs_fop_write_iter+0x335/0x510
[ 990.108050] ? kernfs_ops+0xf0/0xf0
[ 990.112758] new_sync_write+0x333/0x750
[ 990.117857] ? __lock_acquire+0x508/0xc10
[ 990.123154] ? new_sync_read+0x740/0x740
[ 990.128352] ? lock_acquire.part.0+0xbc/0x260
[ 990.134036] ? ksys_write+0x116/0x250
[ 990.138946] vfs_write+0x657/0x9b0
[ 990.143559] ksys_write+0x116/0x250
[ 990.148268] ? __ia32_sys_read+0xc0/0xc0
[ 990.153464] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.159052] ? rcu_is_watching+0x15/0xd0
[ 990.164251] do_syscall_64+0xf3/0x650
[ 990.169156] ? irq_exit_rcu+0x1a/0x30
[ 990.174064] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.180528] RIP: 0033:0x7f0e93afe22e
[ 990.185337] RSP: 002b:00007fff79380960 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.194635] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f0e93afe22e
[ 990.203440] RDX: 0000000000000001 RSI: 000055ea9f20f6b0 RDI: 0000000000000007
[ 990.212263] RBP: 00007fff79380970 R08: 0000000000000000 R09: 0000000000000000
[ 990.221068] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f0e93976390
[ 990.229877] R13: 000055ea9f20f6b0 R14: 0000000000000007 R15: 0000000000000003
[ 990.238692]
[ 990.241948] \x0aShowing all locks held in the system:
[ 990.250486] 1 lock held by khungtaskd/185:
[ 990.255878] #0: ffffffff896fdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.267354] 2 locks held by systemd-journal/802:
[ 990.273333] #0: ff1100180df372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 990.283221] #1: ff1100019cd46378 (&mm->mmap_lock){++++}-{4:4}, at: fs_reclaim_acquire+0x7b/0x120
[ 990.293998] 4 locks held by python3/3770:
[ 990.299294] #0: ff1100012ee143e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.309180] #1: ff11000181312680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.320140] #2: ff11000121a86b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.331296] #3: ff11000121ac9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.341394] 4 locks held by python3/5809:
[ 990.346690] #0: ff1100012ee143e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.356576] #1: ff110001b9ba1480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.367538] #2: ff11000121a86b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.378695] #3: ff11000121ac9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.388776]
[ 990.391242] =============================================\x0a
[ 1112.574164] INFO: task python3:5809 blocked for more than 245 seconds.
[ 1112.582314] Not tainted 7.1.0-rc4-rhie-g7ec97933f2d1 #1
[ 1112.589566] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.599153] task:python3 state:D stack:25536 pid:5809 tgid:5809 ppid:5808 task_flags:0x400100 flags:0x00080000
[ 1112.612356] Call Trace:
[ 1112.615895]
[ 1112.619044] __schedule+0xcc3/0x19c0
[ 1112.623857] ? io_schedule_timeout+0x140/0x140
[ 1112.629639] ? find_held_lock+0x2b/0x80
[ 1112.634812] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.640695] schedule+0xd8/0x260
[ 1112.645110] schedule_preempt_disabled+0x18/0x30
[ 1112.651088] __mutex_lock+0x6af/0x2010
[ 1112.656093] ? threaded_store+0x132/0x220
[ 1112.661391] ? ww_mutex_lock+0x160/0x160
[ 1112.666589] ? selinux_inode_free_security+0x330/0x330
[ 1112.673157] ? cap_capable+0x1d6/0x3e0
[ 1112.678164] ? threaded_store+0x132/0x220
[ 1112.683464] threaded_store+0x132/0x220
[ 1112.688565] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.693862] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.699159] ? sysfs_kf_write+0x65/0x160
[ 1112.704361] ? sysfs_file_ops+0x140/0x140
[ 1112.709656] kernfs_fop_write_iter+0x335/0x510
[ 1112.715438] ? kernfs_ops+0xf0/0xf0
[ 1112.720149] new_sync_write+0x333/0x750
[ 1112.725256] ? __lock_acquire+0x508/0xc10
[ 1112.730551] ? new_sync_read+0x740/0x740
[ 1112.735752] ? lock_acquire.part.0+0xbc/0x260
[ 1112.741437] ? ksys_write+0x116/0x250
[ 1112.746345] vfs_write+0x657/0x9b0
[ 1112.750960] ksys_write+0x116/0x250
[ 1112.755670] ? __ia32_sys_read+0xc0/0xc0
[ 1112.760872] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.766463] ? rcu_is_watching+0x15/0xd0
[ 1112.771666] do_syscall_64+0xf3/0x650
[ 1112.776570] ? trace_hardirqs_off+0xd/0x30
[ 1112.781965] ? exc_page_fault+0xda/0xf0
[ 1112.787067] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.793533] RIP: 0033:0x7fc67b05d22e
[ 1112.798343] RSP: 002b:00007ffd39a890a0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.807651] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc67b05d22e
[ 1112.816462] RDX: 0000000000000001 RSI: 0000562e8695f6b0 RDI: 0000000000000007
[ 1112.825270] RBP: 00007ffd39a890b0 R08: 0000000000000000 R09: 0000000000000000
[ 1112.834083] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc67aed5390
[ 1112.842893] R13: 0000562e8695f6b0 R14: 0000000000000007 R15: 0000000000000003
[ 1112.851710]
[ 1112.851756] INFO: task python3:5809 is blocked on a mutex likely owned by task python3:3770.
[ 1112.865309] task:python3 state:D stack:25672 pid:3770 tgid:3770 ppid:3769 task_flags:0x400100 flags:0x00080802
[ 1112.878513] Call Trace:
[ 1112.882051]
[ 1112.885203] __schedule+0xcc3/0x19c0
[ 1112.890016] ? io_schedule_timeout+0x140/0x140
[ 1112.895800] ? find_held_lock+0x2b/0x80
[ 1112.900903] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.907083] schedule+0xd8/0x260
[ 1112.911501] schedule_timeout+0x13f/0x290
[ 1112.916800] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.923169] ? sysvec_apic_timer_interrupt+0xaa/0xe0
[ 1112.929540] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.935231] ? __free_object+0x2ad/0x490
[ 1112.940433] msleep+0x1b/0x30
[ 1112.944559] napi_set_threaded+0x3a5/0x440
[ 1112.949955] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.955443] netif_set_threaded+0x257/0x400
[ 1112.960936] threaded_store+0x1b8/0x220
[ 1112.966042] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.971338] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.976639] ? sysfs_file_ops+0x140/0x140
[ 1112.981935] kernfs_fop_write_iter+0x335/0x510
[ 1112.987720] ? kernfs_ops+0xf0/0xf0
[ 1112.992434] new_sync_write+0x333/0x750
[ 1112.997537] ? __lock_acquire+0x508/0xc10
[ 1113.002827] ? new_sync_read+0x740/0x740
[ 1113.008027] ? lock_acquire.part.0+0xbc/0x260
[ 1113.013713] ? ksys_write+0x116/0x250
[ 1113.018625] vfs_write+0x657/0x9b0
[ 1113.023241] ksys_write+0x116/0x250
[ 1113.027953] ? __ia32_sys_read+0xc0/0xc0
[ 1113.033145] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.038736] ? rcu_is_watching+0x15/0xd0
[ 1113.043938] do_syscall_64+0xf3/0x650
[ 1113.048845] ? irq_exit_rcu+0x1a/0x30
[ 1113.053752] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.060219] RIP: 0033:0x7f0e93afe22e
[ 1113.065018] RSP: 002b:00007fff79380960 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.074316] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f0e93afe22e
[ 1113.083124] RDX: 0000000000000001 RSI: 000055ea9f20f6b0 RDI: 0000000000000007
[ 1113.091929] RBP: 00007fff79380970 R08: 0000000000000000 R09: 0000000000000000
[ 1113.100738] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f0e93976390
[ 1113.109549] R13: 000055ea9f20f6b0 R14: 0000000000000007 R15: 0000000000000003
[ 1113.118363]
[ 1113.121629] \x0aShowing all locks held in the system:
[ 1113.130165] 1 lock held by khungtaskd/185:
[ 1113.135561] #0: ffffffff896fdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.147030] 3 locks held by systemd-journal/802:
[ 1113.153012] #0: ff1100180e5372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1113.162891] #1: ffffffff899029a0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1113.175711] #2: ff110001e6aa92b0 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 1113.187270] 4 locks held by python3/3770:
[ 1113.192565] #0: ff1100012ee143e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.202453] #1: ff11000181312680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.213410] #2: ff11000121a86b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.224569] #3: ff11000121ac9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.234666] 4 locks held by python3/5809:
[ 1113.239961] #0: ff1100012ee143e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.249849] #1: ff110001b9ba1480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.260811] #2: ff11000121a86b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.271970] #3: ff11000121ac9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.282058] 5 locks held by (systemd-user-)/6856:
[ 1113.288133] #0: ff1100012ef043e8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.298024] #1: ff1100013ef86280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.308988] #2: ffffffff89769b60 (cgroup_mutex){+.+.}-{4:4}, at: cgroup_kn_lock_live+0xe2/0x3e0
[ 1113.319657] #3: ffffffff895686f0 (cpu_hotplug_lock){++++}-{0:0}, at: cgroup_procs_write_start+0x3e5/0x650
[ 1113.331305] #4: ffffffff897698d0 (cgroup_threadgroup_rwsem){++++}-{0:0}, at: cgroup_procs_write_start+0x3f1/0x650
[ 1113.343732]
[ 1113.346200] =============================================\x0a
[ 1174.015684] kworker/u130:2 (349) used greatest stack depth: 21936 bytes left
[ 1235.456287] INFO: task python3:5809 blocked for more than 368 seconds.
[ 1235.464437] Not tainted 7.1.0-rc4-rhie-g7ec97933f2d1 #1
[ 1235.471690] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.481277] task:python3 state:D stack:25536 pid:5809 tgid:5809 ppid:5808 task_flags:0x400100 flags:0x00080002
[ 1235.494483] Call Trace:
[ 1235.498021]
[ 1235.501173] __schedule+0xcc3/0x19c0
[ 1235.505987] ? io_schedule_timeout+0x140/0x140
[ 1235.511764] ? find_held_lock+0x2b/0x80
[ 1235.516866] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.522752] schedule+0xd8/0x260
[ 1235.527168] schedule_preempt_disabled+0x18/0x30
[ 1235.533146] __mutex_lock+0x6af/0x2010
[ 1235.538148] ? threaded_store+0x132/0x220
[ 1235.543448] ? ww_mutex_lock+0x160/0x160
[ 1235.548638] ? selinux_inode_free_security+0x330/0x330
[ 1235.555207] ? cap_capable+0x1d6/0x3e0
[ 1235.560219] ? threaded_store+0x132/0x220
[ 1235.565514] threaded_store+0x132/0x220
[ 1235.570615] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.575908] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.581206] ? sysfs_kf_write+0x65/0x160
[ 1235.586403] ? sysfs_file_ops+0x140/0x140
[ 1235.591696] kernfs_fop_write_iter+0x335/0x510
[ 1235.597478] ? kernfs_ops+0xf0/0xf0
[ 1235.602190] new_sync_write+0x333/0x750
[ 1235.607291] ? __lock_acquire+0x508/0xc10
[ 1235.612586] ? new_sync_read+0x740/0x740
[ 1235.617788] ? lock_acquire.part.0+0xbc/0x260
[ 1235.623471] ? ksys_write+0x116/0x250
[ 1235.628381] vfs_write+0x657/0x9b0
[ 1235.632994] ksys_write+0x116/0x250
[ 1235.637707] ? __ia32_sys_read+0xc0/0xc0
[ 1235.642896] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.648482] ? rcu_is_watching+0x15/0xd0
[ 1235.653684] do_syscall_64+0xf3/0x650
[ 1235.658579] ? trace_hardirqs_off+0xd/0x30
[ 1235.663971] ? exc_page_fault+0xda/0xf0
[ 1235.669072] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.675539] RIP: 0033:0x7fc67b05d22e
[ 1235.680347] RSP: 002b:00007ffd39a890a0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.689645] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc67b05d22e
[ 1235.698453] RDX: 0000000000000001 RSI: 0000562e8695f6b0 RDI: 0000000000000007
[ 1235.707260] RBP: 00007ffd39a890b0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.716067] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc67aed5390
[ 1235.724875] R13: 0000562e8695f6b0 R14: 0000000000000007 R15: 0000000000000003
[ 1235.733686]
[ 1235.736975] INFO: task python3:5809 is blocked on a mutex likely owned by task python3:3770.
[ 1235.747247] task:python3 state:D stack:25672 pid:3770 tgid:3770 ppid:3769 task_flags:0x400100 flags:0x00080802
[ 1235.760450] Call Trace:
[ 1235.763989]
[ 1235.767137] __schedule+0xcc3/0x19c0
[ 1235.771947] ? io_schedule_timeout+0x140/0x140
[ 1235.777730] ? find_held_lock+0x2b/0x80
[ 1235.782832] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.789002] schedule+0xd8/0x260
[ 1235.793422] schedule_timeout+0x13f/0x290
[ 1235.798708] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.805075] ? sysvec_apic_timer_interrupt+0xaa/0xe0
[ 1235.811444] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.817133] ? __free_object+0x2ad/0x490
[ 1235.822333] msleep+0x1b/0x30
[ 1235.826458] napi_set_threaded+0x3a5/0x440
[ 1235.831853] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.837338] netif_set_threaded+0x257/0x400
[ 1235.842831] threaded_store+0x1b8/0x220
[ 1235.847933] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.853229] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.858525] ? sysfs_file_ops+0x140/0x140
[ 1235.863820] kernfs_fop_write_iter+0x335/0x510
[ 1235.869605] ? kernfs_ops+0xf0/0xf0
[ 1235.874313] new_sync_write+0x333/0x750
[ 1235.879414] ? __lock_acquire+0x508/0xc10
[ 1235.884707] ? new_sync_read+0x740/0x740
[ 1235.889906] ? lock_acquire.part.0+0xbc/0x260
[ 1235.895589] ? ksys_write+0x116/0x250
[ 1235.900497] vfs_write+0x657/0x9b0
[ 1235.905102] ksys_write+0x116/0x250
[ 1235.909814] ? __ia32_sys_read+0xc0/0xc0
[ 1235.915013] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.920602] ? rcu_is_watching+0x15/0xd0
[ 1235.925803] do_syscall_64+0xf3/0x650
[ 1235.930706] ? irq_exit_rcu+0x1a/0x30
[ 1235.935613] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.942079] RIP: 0033:0x7f0e93afe22e
[ 1235.946887] RSP: 002b:00007fff79380960 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.956182] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f0e93afe22e
[ 1235.964986] RDX: 0000000000000001 RSI: 000055ea9f20f6b0 RDI: 0000000000000007
[ 1235.973792] RBP: 00007fff79380970 R08: 0000000000000000 R09: 0000000000000000
[ 1235.982596] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f0e93976390
[ 1235.991404] R13: 000055ea9f20f6b0 R14: 0000000000000007 R15: 0000000000000003
[ 1236.000207]
[ 1236.003470] \x0aShowing all locks held in the system:
[ 1236.012009] 1 lock held by khungtaskd/185:
[ 1236.017400] #0: ffffffff896fdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.028876] 4 locks held by python3/3770:
[ 1236.034170] #0: ff1100012ee143e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.044059] #1: ff11000181312680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.055020] #2: ff11000121a86b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.066175] #3: ff11000121ac9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.076268] 4 locks held by python3/5809:
[ 1236.081563] #0: ff1100012ee143e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.091451] #1: ff110001b9ba1480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.102405] #2: ff11000121a86b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.113561] #3: ff11000121ac9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.123650]
[ 1236.126122] =============================================\x0a
[ 1358.337415] INFO: task python3:5809 blocked for more than 491 seconds.
[ 1358.345566] Not tainted 7.1.0-rc4-rhie-g7ec97933f2d1 #1
[ 1358.352817] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.362411] task:python3 state:D stack:25536 pid:5809 tgid:5809 ppid:5808 task_flags:0x400100 flags:0x00080002
[ 1358.375618] Call Trace:
[ 1358.379157]
[ 1358.382310] __schedule+0xcc3/0x19c0
[ 1358.387126] ? io_schedule_timeout+0x140/0x140
[ 1358.392912] ? find_held_lock+0x2b/0x80
[ 1358.398018] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.403903] schedule+0xd8/0x260
[ 1358.408324] schedule_preempt_disabled+0x18/0x30
[ 1358.414306] __mutex_lock+0x6af/0x2010
[ 1358.419309] ? threaded_store+0x132/0x220
[ 1358.424610] ? ww_mutex_lock+0x160/0x160
[ 1358.429808] ? selinux_inode_free_security+0x330/0x330
[ 1358.436368] ? cap_capable+0x1d6/0x3e0
[ 1358.441377] ? threaded_store+0x132/0x220
[ 1358.446674] threaded_store+0x132/0x220
[ 1358.451779] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.457078] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.462377] ? sysfs_kf_write+0x65/0x160
[ 1358.467577] ? sysfs_file_ops+0x140/0x140
[ 1358.472874] kernfs_fop_write_iter+0x335/0x510
[ 1358.478653] ? kernfs_ops+0xf0/0xf0
[ 1358.483366] new_sync_write+0x333/0x750
[ 1358.488471] ? __lock_acquire+0x508/0xc10
[ 1358.493770] ? new_sync_read+0x740/0x740
[ 1358.498972] ? lock_acquire.part.0+0xbc/0x260
[ 1358.504657] ? ksys_write+0x116/0x250
[ 1358.509569] vfs_write+0x657/0x9b0
[ 1358.514185] ksys_write+0x116/0x250
[ 1358.518897] ? __ia32_sys_read+0xc0/0xc0
[ 1358.524091] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.529680] ? rcu_is_watching+0x15/0xd0
[ 1358.534884] do_syscall_64+0xf3/0x650
[ 1358.539792] ? trace_hardirqs_off+0xd/0x30
[ 1358.545188] ? exc_page_fault+0xda/0xf0
[ 1358.550284] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.556752] RIP: 0033:0x7fc67b05d22e
[ 1358.561563] RSP: 002b:00007ffd39a890a0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.570864] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fc67b05d22e
[ 1358.579675] RDX: 0000000000000001 RSI: 0000562e8695f6b0 RDI: 0000000000000007
[ 1358.588483] RBP: 00007ffd39a890b0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.597286] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc67aed5390
[ 1358.606097] R13: 0000562e8695f6b0 R14: 0000000000000007 R15: 0000000000000003
[ 1358.614913]
[ 1358.618228] INFO: task python3:5809 is blocked on a mutex likely owned by task python3:3770.
[ 1358.628500] task:python3 state:D stack:25672 pid:3770 tgid:3770 ppid:3769 task_flags:0x400100 flags:0x00080802
[ 1358.641704] Call Trace:
[ 1358.645249]
[ 1358.648401] __schedule+0xcc3/0x19c0
[ 1358.653215] ? io_schedule_timeout+0x140/0x140
[ 1358.659000] ? find_held_lock+0x2b/0x80
[ 1358.664103] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.670281] schedule+0xd8/0x260
[ 1358.674699] schedule_timeout+0x13f/0x290
[ 1358.679995] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.686363] ? sysvec_apic_timer_interrupt+0xaa/0xe0
[ 1358.692734] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.698423] ? __free_object+0x2ad/0x490
[ 1358.703624] msleep+0x1b/0x30
[ 1358.707749] napi_set_threaded+0x3a5/0x440
[ 1358.713144] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.718633] netif_set_threaded+0x257/0x400
[ 1358.724126] threaded_store+0x1b8/0x220
[ 1358.729232] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.734532] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.739829] ? sysfs_file_ops+0x140/0x140
[ 1358.745127] kernfs_fop_write_iter+0x335/0x510
[ 1358.750914] ? kernfs_ops+0xf0/0xf0
[ 1358.755626] new_sync_write+0x333/0x750
[ 1358.760729] ? __lock_acquire+0x508/0xc10
[ 1358.766026] ? new_sync_read+0x740/0x740
[ 1358.771228] ? lock_acquire.part.0+0xbc/0x260
[ 1358.776915] ? ksys_write+0x116/0x250
[ 1358.781817] vfs_write+0x657/0x9b0
[ 1358.786432] ksys_write+0x116/0x250
[ 1358.791143] ? __ia32_sys_read+0xc0/0xc0
[ 1358.796344] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.801936] ? rcu_is_watching+0x15/0xd0
[ 1358.807205] do_syscall_64+0xf3/0x650
[ 1358.812111] ? irq_exit_rcu+0x1a/0x30
[ 1358.817020] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.823488] RIP: 0033:0x7f0e93afe22e
[ 1358.828287] RSP: 002b:00007fff79380960 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.837584] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f0e93afe22e
[ 1358.846395] RDX: 0000000000000001 RSI: 000055ea9f20f6b0 RDI: 0000000000000007
[ 1358.855204] RBP: 00007fff79380970 R08: 0000000000000000 R09: 0000000000000000
[ 1358.864012] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f0e93976390
[ 1358.872822] R13: 000055ea9f20f6b0 R14: 0000000000000007 R15: 0000000000000003
[ 1358.881639]
[ 1358.884899] \x0aShowing all locks held in the system:
[ 1358.893441] 1 lock held by khungtaskd/185:
[ 1358.898899] #0: ffffffff896fdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.910375] 2 locks held by systemd-journal/802:
[ 1358.916343] #0: ff1100180e3b72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1358.926233] #1: ff1100019cd46378 (&mm->mmap_lock){++++}-{4:4}, at: fs_reclaim_acquire+0x7b/0x120
[ 1358.937018] 4 locks held by python3/3770:
[ 1358.942313] #0: ff1100012ee143e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.952272] #1: ff11000181312680 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.963225] #2: ff11000121a86b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.974384] #3: ff11000121ac9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.984486] 4 locks held by python3/5809:
[ 1358.989784] #0: ff1100012ee143e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.999671] #1: ff110001b9ba1480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1359.010634] #2: ff11000121a86b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.021791] #3: ff11000121ac9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.031871] 1 lock held by systemd/7650:
[ 1359.037072]
[ 1359.039539] =============================================\x0a