[ 789.922923] nipa-hw-worker: drivers/net:napi_threaded.py: timed out [ 789.925259] nipa-hw-worker: [5/46] Retrying drivers/net:napi_threaded.py [ 790.077761] kselftest: Running tests in drivers/net [ 989.719539] INFO: task python3:5807 blocked for more than 122 seconds. [ 989.727779] Not tainted 7.1.0-rc4-wpvm-ga03a277dfcd4 #1 [ 989.735691] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 989.745283] task:python3 state:D stack:25672 pid:5807 tgid:5807 ppid:5806 task_flags:0x400100 flags:0x00080000 [ 989.758491] Call Trace: [ 989.762033] [ 989.765184] __schedule+0xcc3/0x19c0 [ 989.770004] ? io_schedule_timeout+0x140/0x140 [ 989.775791] ? find_held_lock+0x2b/0x80 [ 989.780897] ? __lock_release.isra.0+0x6b/0x1a0 [ 989.786785] schedule+0xd8/0x260 [ 989.791206] schedule_preempt_disabled+0x18/0x30 [ 989.797187] __mutex_lock+0x6af/0x2010 [ 989.802194] ? threaded_store+0x132/0x220 [ 989.807497] ? ww_mutex_lock+0x160/0x160 [ 989.812696] ? selinux_inode_free_security+0x330/0x330 [ 989.819266] ? cap_capable+0x1d6/0x3e0 [ 989.824277] ? threaded_store+0x132/0x220 [ 989.829576] threaded_store+0x132/0x220 [ 989.834676] ? xps_rxqs_store+0x2e0/0x2e0 [ 989.839974] ? sysfs_file_kobj+0xbd/0x1e0 [ 989.845275] ? sysfs_kf_write+0x65/0x160 [ 989.850477] ? sysfs_file_ops+0x140/0x140 [ 989.855775] kernfs_fop_write_iter+0x335/0x510 [ 989.861553] ? kernfs_ops+0xf0/0xf0 [ 989.866265] new_sync_write+0x333/0x750 [ 989.871371] ? __lock_acquire+0x508/0xc10 [ 989.876667] ? new_sync_read+0x740/0x740 [ 989.881869] ? lock_acquire.part.0+0xbc/0x260 [ 989.887556] ? ksys_write+0x116/0x250 [ 989.892457] vfs_write+0x657/0x9b0 [ 989.897075] ksys_write+0x116/0x250 [ 989.901789] ? __ia32_sys_read+0xc0/0xc0 [ 989.906990] ? kernfs_fop_llseek+0x160/0x1f0 [ 989.912581] ? rcu_is_watching+0x15/0xd0 [ 989.917782] do_syscall_64+0xf3/0x650 [ 989.922687] ? trace_hardirqs_off+0xd/0x30 [ 989.928085] ? exc_page_fault+0xda/0xf0 [ 989.933187] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 989.939657] RIP: 0033:0x7f953e63022e [ 989.944475] RSP: 002b:00007fff5dae3db0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 989.953764] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f953e63022e [ 989.962575] RDX: 0000000000000001 RSI: 00005556130996b0 RDI: 0000000000000007 [ 989.971384] RBP: 00007fff5dae3dc0 R08: 0000000000000000 R09: 0000000000000000 [ 989.980193] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f953e4a8390 [ 989.989002] R13: 00005556130996b0 R14: 0000000000000007 R15: 0000000000000003 [ 989.997815] [ 990.001102] INFO: task python3:5807 is blocked on a mutex likely owned by task python3:3778. [ 990.011378] task:python3 state:D stack:25672 pid:3778 tgid:3778 ppid:3777 task_flags:0x400100 flags:0x00080802 [ 990.024581] Call Trace: [ 990.028118] [ 990.031270] __schedule+0xcc3/0x19c0 [ 990.036071] ? io_schedule_timeout+0x140/0x140 [ 990.041845] ? find_held_lock+0x2b/0x80 [ 990.046949] ? __try_to_del_timer_sync+0x155/0x1f0 [ 990.053128] schedule+0xd8/0x260 [ 990.057547] schedule_timeout+0x13f/0x290 [ 990.062844] ? hrtimer_nanosleep_restart+0x270/0x270 [ 990.069213] ? trace_contention_end+0xb3/0x1a0 [ 990.074989] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 990.080673] ? ww_mutex_lock+0x160/0x160 [ 990.085875] msleep+0x1b/0x30 [ 990.090001] napi_set_threaded+0x3a5/0x440 [ 990.095400] ? napi_resume_irqs+0x1a0/0x1a0 [ 990.100894] netif_set_threaded+0x257/0x400 [ 990.106389] threaded_store+0x1b8/0x220 [ 990.111490] ? xps_rxqs_store+0x2e0/0x2e0 [ 990.116788] ? sysfs_file_kobj+0xbd/0x1e0 [ 990.122084] ? sysfs_kf_write+0x65/0x160 [ 990.127283] ? sysfs_file_ops+0x140/0x140 [ 990.132579] kernfs_fop_write_iter+0x335/0x510 [ 990.138366] ? kernfs_ops+0xf0/0xf0 [ 990.143078] new_sync_write+0x333/0x750 [ 990.148181] ? __lock_acquire+0x508/0xc10 [ 990.153472] ? new_sync_read+0x740/0x740 [ 990.158673] ? lock_acquire.part.0+0xbc/0x260 [ 990.164365] ? ksys_write+0x116/0x250 [ 990.169276] vfs_write+0x657/0x9b0 [ 990.173892] ksys_write+0x116/0x250 [ 990.178604] ? __ia32_sys_read+0xc0/0xc0 [ 990.183794] ? kernfs_fop_llseek+0x160/0x1f0 [ 990.189386] ? rcu_is_watching+0x15/0xd0 [ 990.194585] do_syscall_64+0xf3/0x650 [ 990.199490] ? trace_hardirqs_off+0xd/0x30 [ 990.204884] ? exc_page_fault+0xda/0xf0 [ 990.209986] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 990.216455] RIP: 0033:0x7f015075622e [ 990.221265] RSP: 002b:00007fffbbec9b90 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 990.230561] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f015075622e [ 990.239371] RDX: 0000000000000001 RSI: 000055cc3357d6b0 RDI: 0000000000000007 [ 990.248182] RBP: 00007fffbbec9ba0 R08: 0000000000000000 R09: 0000000000000000 [ 990.256990] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f01505ce390 [ 990.265798] R13: 000055cc3357d6b0 R14: 0000000000000007 R15: 0000000000000003 [ 990.274602] [ 990.277863] \x0aShowing all locks held in the system: [ 990.286403] 1 lock held by khungtaskd/185: [ 990.291799] #0: ffffffffadcfdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 990.303271] 3 locks held by systemd-journal/801: [ 990.309252] #0: ff1100180e4372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0 [ 990.319138] #1: ffffffffadf02b20 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120 [ 990.331953] #2: ff1100019f776b20 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0 [ 990.343510] 4 locks held by python3/3778: [ 990.348807] #0: ff1100010593c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.358696] #1: ff1100013d1a6280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.369649] #2: ff11000122706b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.380804] #3: ff11000122769d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.390905] 4 locks held by python3/5807: [ 990.396200] #0: ff1100010593c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.406089] #1: ff110001ef401480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.417053] #2: ff11000122706b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.428211] #3: ff11000122769d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.438299] [ 990.440766] =============================================\x0a [ 1112.600675] INFO: task python3:5807 blocked for more than 245 seconds. [ 1112.608823] Not tainted 7.1.0-rc4-wpvm-ga03a277dfcd4 #1 [ 1112.616074] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1112.625664] task:python3 state:D stack:25672 pid:5807 tgid:5807 ppid:5806 task_flags:0x400100 flags:0x00080000 [ 1112.638862] Call Trace: [ 1112.642400] [ 1112.645548] __schedule+0xcc3/0x19c0 [ 1112.650365] ? io_schedule_timeout+0x140/0x140 [ 1112.656151] ? find_held_lock+0x2b/0x80 [ 1112.661256] ? __lock_release.isra.0+0x6b/0x1a0 [ 1112.667140] schedule+0xd8/0x260 [ 1112.671556] schedule_preempt_disabled+0x18/0x30 [ 1112.677531] __mutex_lock+0x6af/0x2010 [ 1112.682536] ? threaded_store+0x132/0x220 [ 1112.687836] ? ww_mutex_lock+0x160/0x160 [ 1112.693037] ? selinux_inode_free_security+0x330/0x330 [ 1112.699608] ? cap_capable+0x1d6/0x3e0 [ 1112.704618] ? threaded_store+0x132/0x220 [ 1112.704624] threaded_store+0x132/0x220 [ 1112.704628] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.704632] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.725646] ? sysfs_kf_write+0x65/0x160 [ 1112.730851] ? sysfs_file_ops+0x140/0x140 [ 1112.736147] kernfs_fop_write_iter+0x335/0x510 [ 1112.741934] ? kernfs_ops+0xf0/0xf0 [ 1112.746645] new_sync_write+0x333/0x750 [ 1112.751750] ? __lock_acquire+0x508/0xc10 [ 1112.757044] ? new_sync_read+0x740/0x740 [ 1112.762247] ? lock_acquire.part.0+0xbc/0x260 [ 1112.767933] ? ksys_write+0x116/0x250 [ 1112.772844] vfs_write+0x657/0x9b0 [ 1112.777461] ksys_write+0x116/0x250 [ 1112.782174] ? __ia32_sys_read+0xc0/0xc0 [ 1112.787376] ? kernfs_fop_llseek+0x160/0x1f0 [ 1112.792966] ? rcu_is_watching+0x15/0xd0 [ 1112.798170] do_syscall_64+0xf3/0x650 [ 1112.803078] ? trace_hardirqs_off+0xd/0x30 [ 1112.808473] ? exc_page_fault+0xda/0xf0 [ 1112.813576] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1112.820046] RIP: 0033:0x7f953e63022e [ 1112.824855] RSP: 002b:00007fff5dae3db0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1112.834152] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f953e63022e [ 1112.842960] RDX: 0000000000000001 RSI: 00005556130996b0 RDI: 0000000000000007 [ 1112.851769] RBP: 00007fff5dae3dc0 R08: 0000000000000000 R09: 0000000000000000 [ 1112.860577] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f953e4a8390 [ 1112.869385] R13: 00005556130996b0 R14: 0000000000000007 R15: 0000000000000003 [ 1112.878198] [ 1112.881498] INFO: task python3:5807 is blocked on a mutex likely owned by task python3:3778. [ 1112.891771] task:python3 state:D stack:25672 pid:3778 tgid:3778 ppid:3777 task_flags:0x400100 flags:0x00080802 [ 1112.904972] Call Trace: [ 1112.908513] [ 1112.911663] __schedule+0xcc3/0x19c0 [ 1112.916473] ? io_schedule_timeout+0x140/0x140 [ 1112.922249] ? find_held_lock+0x2b/0x80 [ 1112.927352] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1112.933529] schedule+0xd8/0x260 [ 1112.937947] schedule_timeout+0x13f/0x290 [ 1112.943247] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1112.949616] ? trace_contention_end+0xb3/0x1a0 [ 1112.955401] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1112.961091] ? ww_mutex_lock+0x160/0x160 [ 1112.966289] msleep+0x1b/0x30 [ 1112.970414] napi_set_threaded+0x3a5/0x440 [ 1112.975812] ? napi_resume_irqs+0x1a0/0x1a0 [ 1112.981309] netif_set_threaded+0x257/0x400 [ 1112.986802] threaded_store+0x1b8/0x220 [ 1112.991906] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.997203] ? sysfs_file_kobj+0xbd/0x1e0 [ 1113.002500] ? sysfs_kf_write+0x65/0x160 [ 1113.007700] ? sysfs_file_ops+0x140/0x140 [ 1113.012996] kernfs_fop_write_iter+0x335/0x510 [ 1113.018782] ? kernfs_ops+0xf0/0xf0 [ 1113.023496] new_sync_write+0x333/0x750 [ 1113.028599] ? __lock_acquire+0x508/0xc10 [ 1113.033896] ? new_sync_read+0x740/0x740 [ 1113.039087] ? lock_acquire.part.0+0xbc/0x260 [ 1113.044775] ? ksys_write+0x116/0x250 [ 1113.049687] vfs_write+0x657/0x9b0 [ 1113.054303] ksys_write+0x116/0x250 [ 1113.059006] ? __ia32_sys_read+0xc0/0xc0 [ 1113.064198] ? kernfs_fop_llseek+0x160/0x1f0 [ 1113.069786] ? rcu_is_watching+0x15/0xd0 [ 1113.074988] do_syscall_64+0xf3/0x650 [ 1113.079896] ? trace_hardirqs_off+0xd/0x30 [ 1113.085290] ? exc_page_fault+0xda/0xf0 [ 1113.090391] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1113.096857] RIP: 0033:0x7f015075622e [ 1113.101666] RSP: 002b:00007fffbbec9b90 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1113.110960] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f015075622e [ 1113.119767] RDX: 0000000000000001 RSI: 000055cc3357d6b0 RDI: 0000000000000007 [ 1113.128577] RBP: 00007fffbbec9ba0 R08: 0000000000000000 R09: 0000000000000000 [ 1113.137385] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f01505ce390 [ 1113.146191] R13: 000055cc3357d6b0 R14: 0000000000000007 R15: 0000000000000003 [ 1113.155003] [ 1113.158267] \x0aShowing all locks held in the system: [ 1113.166808] 1 lock held by khungtaskd/185: [ 1113.172203] #0: ffffffffadcfdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1113.183678] 4 locks held by python3/3778: [ 1113.188965] #0: ff1100010593c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.198856] #1: ff1100013d1a6280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.209819] #2: ff11000122706b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.220975] #3: ff11000122769d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.231074] 4 locks held by python3/5807: [ 1113.236368] #0: ff1100010593c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.246257] #1: ff110001ef401480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.257220] #2: ff11000122706b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.268380] #3: ff11000122769d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.278469] 5 locks held by (systemd-user-)/6851: [ 1113.284544] #0: ff110001057e43e8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.294435] #1: ff11000145e04480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.305399] #2: ffffffffadd69b60 (cgroup_mutex){+.+.}-{4:4}, at: cgroup_kn_lock_live+0xe2/0x3e0 [ 1113.316066] #3: ffffffffadb686f0 (cpu_hotplug_lock){++++}-{0:0}, at: cgroup_procs_write_start+0x3e5/0x650 [ 1113.327710] #4: ffffffffadd698d0 (cgroup_threadgroup_rwsem){++++}-{0:0}, at: cgroup_procs_write_start+0x3f1/0x650 [ 1113.340130] [ 1113.342596] =============================================\x0a [ 1235.480836] INFO: task python3:5807 blocked for more than 368 seconds. [ 1235.488984] Not tainted 7.1.0-rc4-wpvm-ga03a277dfcd4 #1 [ 1235.496238] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1235.505834] task:python3 state:D stack:25672 pid:5807 tgid:5807 ppid:5806 task_flags:0x400100 flags:0x00080002 [ 1235.519048] Call Trace: [ 1235.522591] [ 1235.525747] __schedule+0xcc3/0x19c0 [ 1235.530568] ? io_schedule_timeout+0x140/0x140 [ 1235.536359] ? find_held_lock+0x2b/0x80 [ 1235.541467] ? __lock_release.isra.0+0x6b/0x1a0 [ 1235.547364] schedule+0xd8/0x260 [ 1235.551787] schedule_preempt_disabled+0x18/0x30 [ 1235.557771] __mutex_lock+0x6af/0x2010 [ 1235.562779] ? threaded_store+0x132/0x220 [ 1235.568084] ? ww_mutex_lock+0x160/0x160 [ 1235.573289] ? selinux_inode_free_security+0x330/0x330 [ 1235.579869] ? cap_capable+0x1d6/0x3e0 [ 1235.584882] ? threaded_store+0x132/0x220 [ 1235.590184] threaded_store+0x132/0x220 [ 1235.595289] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.600592] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.605888] ? sysfs_kf_write+0x65/0x160 [ 1235.611093] ? sysfs_file_ops+0x140/0x140 [ 1235.616394] kernfs_fop_write_iter+0x335/0x510 [ 1235.622183] ? kernfs_ops+0xf0/0xf0 [ 1235.626898] new_sync_write+0x333/0x750 [ 1235.632005] ? __lock_acquire+0x508/0xc10 [ 1235.637304] ? new_sync_read+0x740/0x740 [ 1235.642505] ? lock_acquire.part.0+0xbc/0x260 [ 1235.648196] ? ksys_write+0x116/0x250 [ 1235.653113] vfs_write+0x657/0x9b0 [ 1235.657734] ksys_write+0x116/0x250 [ 1235.662447] ? __ia32_sys_read+0xc0/0xc0 [ 1235.667648] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.673240] ? rcu_is_watching+0x15/0xd0 [ 1235.678444] do_syscall_64+0xf3/0x650 [ 1235.683352] ? trace_hardirqs_off+0xd/0x30 [ 1235.688751] ? exc_page_fault+0xda/0xf0 [ 1235.693860] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.700334] RIP: 0033:0x7f953e63022e [ 1235.705148] RSP: 002b:00007fff5dae3db0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.714449] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f953e63022e [ 1235.723262] RDX: 0000000000000001 RSI: 00005556130996b0 RDI: 0000000000000007 [ 1235.732072] RBP: 00007fff5dae3dc0 R08: 0000000000000000 R09: 0000000000000000 [ 1235.740888] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f953e4a8390 [ 1235.749699] R13: 00005556130996b0 R14: 0000000000000007 R15: 0000000000000003 [ 1235.758524] [ 1235.761816] INFO: task python3:5807 is blocked on a mutex likely owned by task python3:3778. [ 1235.772097] task:python3 state:D stack:25672 pid:3778 tgid:3778 ppid:3777 task_flags:0x400100 flags:0x00080802 [ 1235.785299] Call Trace: [ 1235.788838] [ 1235.791992] __schedule+0xcc3/0x19c0 [ 1235.796810] ? io_schedule_timeout+0x140/0x140 [ 1235.802597] ? find_held_lock+0x2b/0x80 [ 1235.807707] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1235.813890] schedule+0xd8/0x260 [ 1235.818312] schedule_timeout+0x13f/0x290 [ 1235.823614] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1235.829986] ? trace_contention_end+0xb3/0x1a0 [ 1235.835774] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1235.841471] ? ww_mutex_lock+0x160/0x160 [ 1235.846678] msleep+0x1b/0x30 [ 1235.850806] napi_set_threaded+0x3a5/0x440 [ 1235.856205] ? napi_resume_irqs+0x1a0/0x1a0 [ 1235.861709] netif_set_threaded+0x257/0x400 [ 1235.867208] threaded_store+0x1b8/0x220 [ 1235.872316] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.877618] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.882919] ? sysfs_kf_write+0x65/0x160 [ 1235.888122] ? sysfs_file_ops+0x140/0x140 [ 1235.893423] kernfs_fop_write_iter+0x335/0x510 [ 1235.899213] ? kernfs_ops+0xf0/0xf0 [ 1235.903930] new_sync_write+0x333/0x750 [ 1235.909038] ? __lock_acquire+0x508/0xc10 [ 1235.914338] ? new_sync_read+0x740/0x740 [ 1235.919540] ? lock_acquire.part.0+0xbc/0x260 [ 1235.925231] ? ksys_write+0x116/0x250 [ 1235.930144] vfs_write+0x657/0x9b0 [ 1235.934768] ksys_write+0x116/0x250 [ 1235.939485] ? __ia32_sys_read+0xc0/0xc0 [ 1235.944686] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.950276] ? rcu_is_watching+0x15/0xd0 [ 1235.955480] do_syscall_64+0xf3/0x650 [ 1235.960392] ? trace_hardirqs_off+0xd/0x30 [ 1235.965790] ? exc_page_fault+0xda/0xf0 [ 1235.970895] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.977370] RIP: 0033:0x7f015075622e [ 1235.982182] RSP: 002b:00007fffbbec9b90 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.991475] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f015075622e [ 1236.000288] RDX: 0000000000000001 RSI: 000055cc3357d6b0 RDI: 0000000000000007 [ 1236.009102] RBP: 00007fffbbec9ba0 R08: 0000000000000000 R09: 0000000000000000 [ 1236.017913] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f01505ce390 [ 1236.026726] R13: 000055cc3357d6b0 R14: 0000000000000007 R15: 0000000000000003 [ 1236.035543] [ 1236.038813] \x0aShowing all locks held in the system: [ 1236.047362] 1 lock held by khungtaskd/185: [ 1236.052761] #0: ffffffffadcfdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1236.064260] 4 locks held by python3/3778: [ 1236.069559] #0: ff1100010593c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.079457] #1: ff1100013d1a6280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.090427] #2: ff11000122706b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.101592] #3: ff11000122769d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.111706] 4 locks held by python3/5807: [ 1236.117002] #0: ff1100010593c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.126900] #1: ff110001ef401480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.137869] #2: ff11000122706b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.149037] #3: ff11000122769d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.159139] [ 1236.161607] =============================================\x0a [ 1358.363006] INFO: task python3:5807 blocked for more than 491 seconds. [ 1358.371153] Not tainted 7.1.0-rc4-wpvm-ga03a277dfcd4 #1 [ 1358.378395] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1358.387983] task:python3 state:D stack:25672 pid:5807 tgid:5807 ppid:5806 task_flags:0x400100 flags:0x00080002 [ 1358.401188] Call Trace: [ 1358.404728] [ 1358.407880] __schedule+0xcc3/0x19c0 [ 1358.412695] ? io_schedule_timeout+0x140/0x140 [ 1358.418481] ? find_held_lock+0x2b/0x80 [ 1358.423587] ? __lock_release.isra.0+0x6b/0x1a0 [ 1358.429487] schedule+0xd8/0x260 [ 1358.433901] schedule_preempt_disabled+0x18/0x30 [ 1358.439885] __mutex_lock+0x6af/0x2010 [ 1358.444890] ? threaded_store+0x132/0x220 [ 1358.450180] ? ww_mutex_lock+0x160/0x160 [ 1358.455379] ? selinux_inode_free_security+0x330/0x330 [ 1358.461950] ? cap_capable+0x1d6/0x3e0 [ 1358.466957] ? threaded_store+0x132/0x220 [ 1358.472255] threaded_store+0x132/0x220 [ 1358.477355] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.482653] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.487951] ? sysfs_kf_write+0x65/0x160 [ 1358.493149] ? sysfs_file_ops+0x140/0x140 [ 1358.498444] kernfs_fop_write_iter+0x335/0x510 [ 1358.504230] ? kernfs_ops+0xf0/0xf0 [ 1358.508942] new_sync_write+0x333/0x750 [ 1358.514046] ? __lock_acquire+0x508/0xc10 [ 1358.519344] ? new_sync_read+0x740/0x740 [ 1358.524547] ? lock_acquire.part.0+0xbc/0x260 [ 1358.530231] ? ksys_write+0x116/0x250 [ 1358.535139] vfs_write+0x657/0x9b0 [ 1358.539743] ksys_write+0x116/0x250 [ 1358.544455] ? __ia32_sys_read+0xc0/0xc0 [ 1358.549645] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.555232] ? rcu_is_watching+0x15/0xd0 [ 1358.560434] do_syscall_64+0xf3/0x650 [ 1358.565332] ? trace_hardirqs_off+0xd/0x30 [ 1358.570727] ? exc_page_fault+0xda/0xf0 [ 1358.575835] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.582301] RIP: 0033:0x7f953e63022e [ 1358.587101] RSP: 002b:00007fff5dae3db0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.596397] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f953e63022e [ 1358.605206] RDX: 0000000000000001 RSI: 00005556130996b0 RDI: 0000000000000007 [ 1358.614014] RBP: 00007fff5dae3dc0 R08: 0000000000000000 R09: 0000000000000000 [ 1358.622823] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f953e4a8390 [ 1358.631633] R13: 00005556130996b0 R14: 0000000000000007 R15: 0000000000000003 [ 1358.640444] [ 1358.643719] INFO: task python3:5807 is blocked on a mutex likely owned by task python3:3778. [ 1358.653992] task:python3 state:D stack:25672 pid:3778 tgid:3778 ppid:3777 task_flags:0x400100 flags:0x00080802 [ 1358.667193] Call Trace: [ 1358.670730] [ 1358.673879] __schedule+0xcc3/0x19c0 [ 1358.678689] ? io_schedule_timeout+0x140/0x140 [ 1358.684465] ? find_held_lock+0x2b/0x80 [ 1358.689567] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1358.695746] schedule+0xd8/0x260 [ 1358.700164] schedule_timeout+0x13f/0x290 [ 1358.705462] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1358.711834] ? trace_contention_end+0xb3/0x1a0 [ 1358.717620] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1358.723308] ? ww_mutex_lock+0x160/0x160 [ 1358.728496] msleep+0x1b/0x30 [ 1358.732621] napi_set_threaded+0x3a5/0x440 [ 1358.738016] ? napi_resume_irqs+0x1a0/0x1a0 [ 1358.743508] netif_set_threaded+0x257/0x400 [ 1358.749002] threaded_store+0x1b8/0x220 [ 1358.754104] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.759404] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.764698] ? sysfs_kf_write+0x65/0x160 [ 1358.769896] ? sysfs_file_ops+0x140/0x140 [ 1358.775192] kernfs_fop_write_iter+0x335/0x510 [ 1358.780978] ? kernfs_ops+0xf0/0xf0 [ 1358.785687] new_sync_write+0x333/0x750 [ 1358.790879] ? __lock_acquire+0x508/0xc10 [ 1358.796176] ? new_sync_read+0x740/0x740 [ 1358.801375] ? lock_acquire.part.0+0xbc/0x260 [ 1358.807061] ? ksys_write+0x116/0x250 [ 1358.811971] vfs_write+0x657/0x9b0 [ 1358.816588] ksys_write+0x116/0x250 [ 1358.821300] ? __ia32_sys_read+0xc0/0xc0 [ 1358.826500] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.832080] ? rcu_is_watching+0x15/0xd0 [ 1358.837280] do_syscall_64+0xf3/0x650 [ 1358.842179] ? trace_hardirqs_off+0xd/0x30 [ 1358.847562] ? exc_page_fault+0xda/0xf0 [ 1358.852662] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.859129] RIP: 0033:0x7f015075622e [ 1358.863937] RSP: 002b:00007fffbbec9b90 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.873236] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f015075622e [ 1358.882044] RDX: 0000000000000001 RSI: 000055cc3357d6b0 RDI: 0000000000000007 [ 1358.890843] RBP: 00007fffbbec9ba0 R08: 0000000000000000 R09: 0000000000000000 [ 1358.899651] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f01505ce390 [ 1358.908458] R13: 000055cc3357d6b0 R14: 0000000000000007 R15: 0000000000000003 [ 1358.917272] [ 1358.920538] \x0aShowing all locks held in the system: [ 1358.929076] 1 lock held by khungtaskd/185: [ 1358.934469] #0: ffffffffadcfdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1358.945950] 4 locks held by python3/3778: [ 1358.951247] #0: ff1100010593c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1358.961135] #1: ff1100013d1a6280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1358.972095] #2: ff11000122706b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1358.983249] #3: ff11000122769d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1358.993343] 4 locks held by python3/5807: [ 1358.998638] #0: ff1100010593c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1359.008527] #1: ff110001ef401480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1359.019487] #2: ff11000122706b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1359.030643] #3: ff11000122769d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1359.040733] 1 lock held by systemd/7646: [ 1359.045931] [ 1359.048398] =============================================\x0a