[ 787.330626] nipa-hw-worker: drivers/net:napi_threaded.py: timed out [ 787.332988] nipa-hw-worker: [5/46] Retrying drivers/net:napi_threaded.py [ 787.494879] kselftest: Running tests in drivers/net [ 989.686306] INFO: task python3:5815 blocked for more than 122 seconds. [ 989.694522] Not tainted 7.1.0-rc4-buzj-g554a61d6e749 #1 [ 989.701773] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 989.711364] task:python3 state:D stack:25520 pid:5815 tgid:5815 ppid:5814 task_flags:0x400100 flags:0x00080000 [ 989.725064] Call Trace: [ 989.728604] [ 989.731754] __schedule+0xcc3/0x19c0 [ 989.736570] ? io_schedule_timeout+0x140/0x140 [ 989.742355] ? find_held_lock+0x2b/0x80 [ 989.747457] ? __lock_release.isra.0+0x6b/0x1a0 [ 989.753345] schedule+0xd8/0x260 [ 989.757760] schedule_preempt_disabled+0x18/0x30 [ 989.763740] __mutex_lock+0x6af/0x2010 [ 989.768737] ? threaded_store+0x132/0x220 [ 989.774034] ? ww_mutex_lock+0x160/0x160 [ 989.779231] ? selinux_inode_free_security+0x330/0x330 [ 989.785792] ? cap_capable+0x1d6/0x3e0 [ 989.790799] ? threaded_store+0x132/0x220 [ 989.796096] threaded_store+0x132/0x220 [ 989.801197] ? xps_rxqs_store+0x2e0/0x2e0 [ 989.806494] ? sysfs_file_kobj+0xbd/0x1e0 [ 989.811792] ? sysfs_kf_write+0x65/0x160 [ 989.816992] ? sysfs_file_ops+0x140/0x140 [ 989.822288] kernfs_fop_write_iter+0x335/0x510 [ 989.828071] ? kernfs_ops+0xf0/0xf0 [ 989.832782] new_sync_write+0x333/0x750 [ 989.837874] ? __lock_acquire+0x508/0xc10 [ 989.843169] ? new_sync_read+0x740/0x740 [ 989.848366] ? lock_acquire.part.0+0xbc/0x260 [ 989.854049] ? ksys_write+0x116/0x250 [ 989.858958] vfs_write+0x657/0x9b0 [ 989.863574] ksys_write+0x116/0x250 [ 989.868283] ? __ia32_sys_read+0xc0/0xc0 [ 989.873480] ? kernfs_fop_llseek+0x160/0x1f0 [ 989.879065] ? rcu_is_watching+0x15/0xd0 [ 989.884267] do_syscall_64+0xf3/0x650 [ 989.889174] ? irq_exit_rcu+0x1a/0x30 [ 989.894071] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 989.900540] RIP: 0033:0x7fca1d07022e [ 989.905347] RSP: 002b:00007ffde3aa6a40 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 989.914645] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fca1d07022e [ 989.923453] RDX: 0000000000000001 RSI: 0000557aab4546b0 RDI: 0000000000000007 [ 989.932250] RBP: 00007ffde3aa6a50 R08: 0000000000000000 R09: 0000000000000000 [ 989.941056] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fca1cee8390 [ 989.949862] R13: 0000557aab4546b0 R14: 0000000000000007 R15: 0000000000000003 [ 989.958673] [ 989.961949] INFO: task python3:5815 is blocked on a mutex likely owned by task python3:3751. [ 989.972220] task:python3 state:D stack:25672 pid:3751 tgid:3751 ppid:3750 task_flags:0x400100 flags:0x00080802 [ 989.985419] Call Trace: [ 989.988956] [ 989.992103] __schedule+0xcc3/0x19c0 [ 989.996911] ? io_schedule_timeout+0x140/0x140 [ 990.002693] ? find_held_lock+0x2b/0x80 [ 990.007794] ? __try_to_del_timer_sync+0x155/0x1f0 [ 990.013974] schedule+0xd8/0x260 [ 990.018393] schedule_timeout+0x13f/0x290 [ 990.023691] ? hrtimer_nanosleep_restart+0x270/0x270 [ 990.030058] ? trace_contention_end+0xb3/0x1a0 [ 990.035840] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 990.041529] ? ww_mutex_lock+0x160/0x160 [ 990.046726] msleep+0x1b/0x30 [ 990.050852] napi_set_threaded+0x3a5/0x440 [ 990.056247] ? napi_resume_irqs+0x1a0/0x1a0 [ 990.061741] netif_set_threaded+0x257/0x400 [ 990.067235] threaded_store+0x1b8/0x220 [ 990.072336] ? xps_rxqs_store+0x2e0/0x2e0 [ 990.077634] ? sysfs_file_kobj+0xbd/0x1e0 [ 990.082921] ? sysfs_kf_write+0x65/0x160 [ 990.088119] ? sysfs_file_ops+0x140/0x140 [ 990.093414] kernfs_fop_write_iter+0x335/0x510 [ 990.099199] ? kernfs_ops+0xf0/0xf0 [ 990.103911] new_sync_write+0x333/0x750 [ 990.109010] ? __lock_acquire+0x508/0xc10 [ 990.114306] ? new_sync_read+0x740/0x740 [ 990.119505] ? lock_acquire.part.0+0xbc/0x260 [ 990.125189] ? ksys_write+0x116/0x250 [ 990.130096] vfs_write+0x657/0x9b0 [ 990.134702] ksys_write+0x116/0x250 [ 990.139412] ? __ia32_sys_read+0xc0/0xc0 [ 990.144611] ? kernfs_fop_llseek+0x160/0x1f0 [ 990.150201] ? rcu_is_watching+0x15/0xd0 [ 990.155400] do_syscall_64+0xf3/0x650 [ 990.160306] ? irq_exit_rcu+0x1a/0x30 [ 990.165214] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 990.171672] RIP: 0033:0x7fa24723922e [ 990.176478] RSP: 002b:00007fff4d5c61c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 990.185773] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa24723922e [ 990.194581] RDX: 0000000000000001 RSI: 000055ed540586b0 RDI: 0000000000000007 [ 990.203388] RBP: 00007fff4d5c61d0 R08: 0000000000000000 R09: 0000000000000000 [ 990.212188] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa2470b1390 [ 990.220995] R13: 000055ed540586b0 R14: 0000000000000007 R15: 0000000000000003 [ 990.229807] [ 990.233063] \x0aShowing all locks held in the system: [ 990.241599] 1 lock held by khungtaskd/185: [ 990.246994] #0: ffffffff8b0fdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 990.258465] 3 locks held by systemd-journal/802: [ 990.264443] #0: ff1100180dfb72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0 [ 990.274330] #1: ff11000196a76378 (&mm->mmap_lock){++++}-{4:4}, at: fs_reclaim_acquire+0x7b/0x120 [ 990.285089] #2: ff11000193ea8020 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0 [ 990.296649] 4 locks held by python3/3751: [ 990.301943] #0: ff110001297a43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.311831] #1: ff1100016f920280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.322793] #2: ff11000123e44b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.333937] #3: ff11000123e89d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.344030] 4 locks held by python3/5815: [ 990.349326] #0: ff110001297a43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.359213] #1: ff110001736b2c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.370172] #2: ff11000123e44b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.381327] #3: ff11000123e89d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.391412] [ 990.393876] =============================================\x0a [ 1112.567515] INFO: task python3:5815 blocked for more than 245 seconds. [ 1112.575670] Not tainted 7.1.0-rc4-buzj-g554a61d6e749 #1 [ 1112.582919] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1112.592511] task:python3 state:D stack:25520 pid:5815 tgid:5815 ppid:5814 task_flags:0x400100 flags:0x00080000 [ 1112.605719] Call Trace: [ 1112.609261] [ 1112.612414] __schedule+0xcc3/0x19c0 [ 1112.617232] ? io_schedule_timeout+0x140/0x140 [ 1112.623018] ? find_held_lock+0x2b/0x80 [ 1112.628125] ? __lock_release.isra.0+0x6b/0x1a0 [ 1112.634014] schedule+0xd8/0x260 [ 1112.638435] schedule_preempt_disabled+0x18/0x30 [ 1112.644415] __mutex_lock+0x6af/0x2010 [ 1112.649423] ? threaded_store+0x132/0x220 [ 1112.654724] ? ww_mutex_lock+0x160/0x160 [ 1112.659926] ? selinux_inode_free_security+0x330/0x330 [ 1112.666499] ? cap_capable+0x1d6/0x3e0 [ 1112.671506] ? threaded_store+0x132/0x220 [ 1112.676804] threaded_store+0x132/0x220 [ 1112.681985] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.687286] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.692585] ? sysfs_kf_write+0x65/0x160 [ 1112.697778] ? sysfs_file_ops+0x140/0x140 [ 1112.703074] kernfs_fop_write_iter+0x335/0x510 [ 1112.708861] ? kernfs_ops+0xf0/0xf0 [ 1112.713577] new_sync_write+0x333/0x750 [ 1112.718681] ? __lock_acquire+0x508/0xc10 [ 1112.723972] ? new_sync_read+0x740/0x740 [ 1112.729178] ? lock_acquire.part.0+0xbc/0x260 [ 1112.734869] ? ksys_write+0x116/0x250 [ 1112.739781] vfs_write+0x657/0x9b0 [ 1112.744399] ksys_write+0x116/0x250 [ 1112.749113] ? __ia32_sys_read+0xc0/0xc0 [ 1112.754303] ? kernfs_fop_llseek+0x160/0x1f0 [ 1112.759892] ? rcu_is_watching+0x15/0xd0 [ 1112.765087] do_syscall_64+0xf3/0x650 [ 1112.769994] ? irq_exit_rcu+0x1a/0x30 [ 1112.774896] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1112.781366] RIP: 0033:0x7fca1d07022e [ 1112.786179] RSP: 002b:00007ffde3aa6a40 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1112.795476] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fca1d07022e [ 1112.804284] RDX: 0000000000000001 RSI: 0000557aab4546b0 RDI: 0000000000000007 [ 1112.813163] RBP: 00007ffde3aa6a50 R08: 0000000000000000 R09: 0000000000000000 [ 1112.821971] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fca1cee8390 [ 1112.830783] R13: 0000557aab4546b0 R14: 0000000000000007 R15: 0000000000000003 [ 1112.839598] [ 1112.842879] INFO: task python3:5815 is blocked on a mutex likely owned by task python3:3751. [ 1112.853153] task:python3 state:D stack:25672 pid:3751 tgid:3751 ppid:3750 task_flags:0x400100 flags:0x00080802 [ 1112.866358] Call Trace: [ 1112.869896] [ 1112.873048] __schedule+0xcc3/0x19c0 [ 1112.877860] ? io_schedule_timeout+0x140/0x140 [ 1112.883646] ? find_held_lock+0x2b/0x80 [ 1112.888749] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1112.894933] schedule+0xd8/0x260 [ 1112.899351] schedule_timeout+0x13f/0x290 [ 1112.904650] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1112.911020] ? trace_contention_end+0xb3/0x1a0 [ 1112.916806] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1112.922499] ? ww_mutex_lock+0x160/0x160 [ 1112.927698] msleep+0x1b/0x30 [ 1112.931826] napi_set_threaded+0x3a5/0x440 [ 1112.937224] ? napi_resume_irqs+0x1a0/0x1a0 [ 1112.942719] netif_set_threaded+0x257/0x400 [ 1112.948215] threaded_store+0x1b8/0x220 [ 1112.953318] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.958619] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.963917] ? sysfs_kf_write+0x65/0x160 [ 1112.969172] ? sysfs_file_ops+0x140/0x140 [ 1112.974468] kernfs_fop_write_iter+0x335/0x510 [ 1112.980318] ? kernfs_ops+0xf0/0xf0 [ 1112.985033] new_sync_write+0x333/0x750 [ 1112.990138] ? __lock_acquire+0x508/0xc10 [ 1112.995433] ? new_sync_read+0x740/0x740 [ 1113.000634] ? lock_acquire.part.0+0xbc/0x260 [ 1113.006322] ? ksys_write+0x116/0x250 [ 1113.011232] vfs_write+0x657/0x9b0 [ 1113.015850] ksys_write+0x116/0x250 [ 1113.020565] ? __ia32_sys_read+0xc0/0xc0 [ 1113.025764] ? kernfs_fop_llseek+0x160/0x1f0 [ 1113.031355] ? rcu_is_watching+0x15/0xd0 [ 1113.036555] do_syscall_64+0xf3/0x650 [ 1113.041456] ? irq_exit_rcu+0x1a/0x30 [ 1113.046363] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1113.052831] RIP: 0033:0x7fa24723922e [ 1113.057642] RSP: 002b:00007fff4d5c61c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1113.066941] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa24723922e [ 1113.075750] RDX: 0000000000000001 RSI: 000055ed540586b0 RDI: 0000000000000007 [ 1113.084558] RBP: 00007fff4d5c61d0 R08: 0000000000000000 R09: 0000000000000000 [ 1113.093370] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa2470b1390 [ 1113.102249] R13: 000055ed540586b0 R14: 0000000000000007 R15: 0000000000000003 [ 1113.111135] [ 1113.114398] \x0aShowing all locks held in the system: [ 1113.122940] 1 lock held by khungtaskd/185: [ 1113.128332] #0: ffffffff8b0fdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1113.140220] 3 locks held by systemd-journal/802: [ 1113.146200] #0: ff1100180ddb72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0 [ 1113.156088] #1: ff11000196a76378 (&mm->mmap_lock){++++}-{4:4}, at: fs_reclaim_acquire+0x7b/0x120 [ 1113.166859] #2: ff1100019a9f5e60 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0 [ 1113.178422] 4 locks held by python3/3751: [ 1113.183721] #0: ff110001297a43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.193610] #1: ff1100016f920280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.204564] #2: ff11000123e44b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.215722] #3: ff11000123e89d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.225818] 4 locks held by python3/5815: [ 1113.231115] #0: ff110001297a43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.241006] #1: ff110001736b2c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.251966] #2: ff11000123e44b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.263123] #3: ff11000123e89d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.273214] 2 locks held by systemd/6863: [ 1113.278526] [ 1113.280996] =============================================\x0a [ 1235.447705] INFO: task python3:5815 blocked for more than 368 seconds. [ 1235.455853] Not tainted 7.1.0-rc4-buzj-g554a61d6e749 #1 [ 1235.463103] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1235.472690] task:python3 state:D stack:25520 pid:5815 tgid:5815 ppid:5814 task_flags:0x400100 flags:0x00080002 [ 1235.485894] Call Trace: [ 1235.489434] [ 1235.492583] __schedule+0xcc3/0x19c0 [ 1235.497398] ? io_schedule_timeout+0x140/0x140 [ 1235.503175] ? find_held_lock+0x2b/0x80 [ 1235.508277] ? __lock_release.isra.0+0x6b/0x1a0 [ 1235.514161] schedule+0xd8/0x260 [ 1235.518577] schedule_preempt_disabled+0x18/0x30 [ 1235.524556] __mutex_lock+0x6af/0x2010 [ 1235.529559] ? threaded_store+0x132/0x220 [ 1235.534849] ? ww_mutex_lock+0x160/0x160 [ 1235.540049] ? selinux_inode_free_security+0x330/0x330 [ 1235.546619] ? cap_capable+0x1d6/0x3e0 [ 1235.551627] ? threaded_store+0x132/0x220 [ 1235.556922] threaded_store+0x132/0x220 [ 1235.562025] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.567310] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.572610] ? sysfs_kf_write+0x65/0x160 [ 1235.577809] ? sysfs_file_ops+0x140/0x140 [ 1235.583103] kernfs_fop_write_iter+0x335/0x510 [ 1235.588888] ? kernfs_ops+0xf0/0xf0 [ 1235.593600] new_sync_write+0x333/0x750 [ 1235.598701] ? __lock_acquire+0x508/0xc10 [ 1235.603998] ? new_sync_read+0x740/0x740 [ 1235.609197] ? lock_acquire.part.0+0xbc/0x260 [ 1235.614880] ? ksys_write+0x116/0x250 [ 1235.619789] vfs_write+0x657/0x9b0 [ 1235.624403] ksys_write+0x116/0x250 [ 1235.629112] ? __ia32_sys_read+0xc0/0xc0 [ 1235.634310] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.639897] ? rcu_is_watching+0x15/0xd0 [ 1235.645098] do_syscall_64+0xf3/0x650 [ 1235.649994] ? irq_exit_rcu+0x1a/0x30 [ 1235.654899] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.661366] RIP: 0033:0x7fca1d07022e [ 1235.666174] RSP: 002b:00007ffde3aa6a40 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.675472] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fca1d07022e [ 1235.684278] RDX: 0000000000000001 RSI: 0000557aab4546b0 RDI: 0000000000000007 [ 1235.693085] RBP: 00007ffde3aa6a50 R08: 0000000000000000 R09: 0000000000000000 [ 1235.701890] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fca1cee8390 [ 1235.710698] R13: 0000557aab4546b0 R14: 0000000000000007 R15: 0000000000000003 [ 1235.719501] [ 1235.722781] INFO: task python3:5815 is blocked on a mutex likely owned by task python3:3751. [ 1235.733054] task:python3 state:D stack:25672 pid:3751 tgid:3751 ppid:3750 task_flags:0x400100 flags:0x00080802 [ 1235.746256] Call Trace: [ 1235.749791] [ 1235.752940] __schedule+0xcc3/0x19c0 [ 1235.757749] ? io_schedule_timeout+0x140/0x140 [ 1235.763531] ? find_held_lock+0x2b/0x80 [ 1235.768633] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1235.774809] schedule+0xd8/0x260 [ 1235.779226] schedule_timeout+0x13f/0x290 [ 1235.784519] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1235.790887] ? trace_contention_end+0xb3/0x1a0 [ 1235.796669] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1235.802359] ? ww_mutex_lock+0x160/0x160 [ 1235.807559] msleep+0x1b/0x30 [ 1235.811675] napi_set_threaded+0x3a5/0x440 [ 1235.817071] ? napi_resume_irqs+0x1a0/0x1a0 [ 1235.822564] netif_set_threaded+0x257/0x400 [ 1235.828057] threaded_store+0x1b8/0x220 [ 1235.833160] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.838447] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.843744] ? sysfs_kf_write+0x65/0x160 [ 1235.848943] ? sysfs_file_ops+0x140/0x140 [ 1235.854237] kernfs_fop_write_iter+0x335/0x510 [ 1235.860022] ? kernfs_ops+0xf0/0xf0 [ 1235.864730] new_sync_write+0x333/0x750 [ 1235.869831] ? __lock_acquire+0x508/0xc10 [ 1235.875125] ? new_sync_read+0x740/0x740 [ 1235.880325] ? lock_acquire.part.0+0xbc/0x260 [ 1235.886009] ? ksys_write+0x116/0x250 [ 1235.890915] vfs_write+0x657/0x9b0 [ 1235.895530] ksys_write+0x116/0x250 [ 1235.900242] ? __ia32_sys_read+0xc0/0xc0 [ 1235.905441] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.911029] ? rcu_is_watching+0x15/0xd0 [ 1235.916226] do_syscall_64+0xf3/0x650 [ 1235.921131] ? irq_exit_rcu+0x1a/0x30 [ 1235.926036] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.932503] RIP: 0033:0x7fa24723922e [ 1235.937310] RSP: 002b:00007fff4d5c61c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.946605] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa24723922e [ 1235.955412] RDX: 0000000000000001 RSI: 000055ed540586b0 RDI: 0000000000000007 [ 1235.964218] RBP: 00007fff4d5c61d0 R08: 0000000000000000 R09: 0000000000000000 [ 1235.973026] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa2470b1390 [ 1235.981832] R13: 000055ed540586b0 R14: 0000000000000007 R15: 0000000000000003 [ 1235.990646] [ 1235.993910] \x0aShowing all locks held in the system: [ 1236.002448] 1 lock held by khungtaskd/185: [ 1236.007842] #0: ffffffff8b0fdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1236.019323] 4 locks held by python3/3751: [ 1236.024618] #0: ff110001297a43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.034508] #1: ff1100016f920280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.045469] #2: ff11000123e44b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.056622] #3: ff11000123e89d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.066716] 4 locks held by python3/5815: [ 1236.072011] #0: ff110001297a43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.081900] #1: ff110001736b2c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.092860] #2: ff11000123e44b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.104019] #3: ff11000123e89d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.114110] [ 1236.116566] =============================================\x0a [ 1358.328930] INFO: task python3:5815 blocked for more than 491 seconds. [ 1358.337082] Not tainted 7.1.0-rc4-buzj-g554a61d6e749 #1 [ 1358.344334] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1358.353926] task:python3 state:D stack:25520 pid:5815 tgid:5815 ppid:5814 task_flags:0x400100 flags:0x00080002 [ 1358.367131] Call Trace: [ 1358.370670] [ 1358.373817] __schedule+0xcc3/0x19c0 [ 1358.378631] ? io_schedule_timeout+0x140/0x140 [ 1358.384417] ? find_held_lock+0x2b/0x80 [ 1358.389520] ? __lock_release.isra.0+0x6b/0x1a0 [ 1358.395404] schedule+0xd8/0x260 [ 1358.399820] schedule_preempt_disabled+0x18/0x30 [ 1358.405797] __mutex_lock+0x6af/0x2010 [ 1358.410799] ? threaded_store+0x132/0x220 [ 1358.416099] ? ww_mutex_lock+0x160/0x160 [ 1358.421359] ? selinux_inode_free_security+0x330/0x330 [ 1358.427930] ? cap_capable+0x1d6/0x3e0 [ 1358.432935] ? threaded_store+0x132/0x220 [ 1358.438231] threaded_store+0x132/0x220 [ 1358.443330] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.448627] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.453927] ? sysfs_kf_write+0x65/0x160 [ 1358.459126] ? sysfs_file_ops+0x140/0x140 [ 1358.464422] kernfs_fop_write_iter+0x335/0x510 [ 1358.470204] ? kernfs_ops+0xf0/0xf0 [ 1358.474914] new_sync_write+0x333/0x750 [ 1358.480010] ? __lock_acquire+0x508/0xc10 [ 1358.485303] ? new_sync_read+0x740/0x740 [ 1358.490504] ? lock_acquire.part.0+0xbc/0x260 [ 1358.496180] ? ksys_write+0x116/0x250 [ 1358.501088] vfs_write+0x657/0x9b0 [ 1358.505695] ksys_write+0x116/0x250 [ 1358.510407] ? __ia32_sys_read+0xc0/0xc0 [ 1358.515605] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.521195] ? rcu_is_watching+0x15/0xd0 [ 1358.526398] do_syscall_64+0xf3/0x650 [ 1358.531303] ? irq_exit_rcu+0x1a/0x30 [ 1358.536209] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.542678] RIP: 0033:0x7fca1d07022e [ 1358.547486] RSP: 002b:00007ffde3aa6a40 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.556784] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fca1d07022e [ 1358.565583] RDX: 0000000000000001 RSI: 0000557aab4546b0 RDI: 0000000000000007 [ 1358.574391] RBP: 00007ffde3aa6a50 R08: 0000000000000000 R09: 0000000000000000 [ 1358.583199] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fca1cee8390 [ 1358.592005] R13: 0000557aab4546b0 R14: 0000000000000007 R15: 0000000000000003 [ 1358.600816] [ 1358.604109] INFO: task python3:5815 is blocked on a mutex likely owned by task python3:3751. [ 1358.614374] task:python3 state:D stack:25672 pid:3751 tgid:3751 ppid:3750 task_flags:0x400100 flags:0x00080802 [ 1358.627576] Call Trace: [ 1358.631112] [ 1358.634261] __schedule+0xcc3/0x19c0 [ 1358.639072] ? io_schedule_timeout+0x140/0x140 [ 1358.644847] ? find_held_lock+0x2b/0x80 [ 1358.649948] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1358.656127] schedule+0xd8/0x260 [ 1358.660545] schedule_timeout+0x13f/0x290 [ 1358.665843] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1358.672204] ? trace_contention_end+0xb3/0x1a0 [ 1358.677988] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1358.683679] ? ww_mutex_lock+0x160/0x160 [ 1358.688878] msleep+0x1b/0x30 [ 1358.693004] napi_set_threaded+0x3a5/0x440 [ 1358.698398] ? napi_resume_irqs+0x1a0/0x1a0 [ 1358.703957] netif_set_threaded+0x257/0x400 [ 1358.709452] threaded_store+0x1b8/0x220 [ 1358.714556] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.719850] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.725145] ? sysfs_kf_write+0x65/0x160 [ 1358.730345] ? sysfs_file_ops+0x140/0x140 [ 1358.735639] kernfs_fop_write_iter+0x335/0x510 [ 1358.741424] ? kernfs_ops+0xf0/0xf0 [ 1358.746136] new_sync_write+0x333/0x750 [ 1358.751229] ? __lock_acquire+0x508/0xc10 [ 1358.756522] ? new_sync_read+0x740/0x740 [ 1358.761721] ? lock_acquire.part.0+0xbc/0x260 [ 1358.767408] ? ksys_write+0x116/0x250 [ 1358.772308] vfs_write+0x657/0x9b0 [ 1358.776920] ksys_write+0x116/0x250 [ 1358.781632] ? __ia32_sys_read+0xc0/0xc0 [ 1358.786894] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.792482] ? rcu_is_watching+0x15/0xd0 [ 1358.797681] do_syscall_64+0xf3/0x650 [ 1358.802587] ? irq_exit_rcu+0x1a/0x30 [ 1358.807495] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.813952] RIP: 0033:0x7fa24723922e [ 1358.818763] RSP: 002b:00007fff4d5c61c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.828060] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa24723922e [ 1358.836868] RDX: 0000000000000001 RSI: 000055ed540586b0 RDI: 0000000000000007 [ 1358.845676] RBP: 00007fff4d5c61d0 R08: 0000000000000000 R09: 0000000000000000 [ 1358.854471] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fa2470b1390 [ 1358.863277] R13: 000055ed540586b0 R14: 0000000000000007 R15: 0000000000000003 [ 1358.872089] [ 1358.875355] \x0aShowing all locks held in the system: [ 1358.883887] 1 lock held by khungtaskd/185: [ 1358.889282] #0: ffffffff8b0fdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1358.900754] 3 locks held by systemd-journal/802: [ 1358.906729] #0: ff1100180df372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0 [ 1358.916616] #1: ffffffff8b3029a0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120 [ 1358.929425] #2: ff1100019a9f5e60 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0 [ 1358.940985] 4 locks held by python3/3751: [ 1358.946278] #0: ff110001297a43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1358.956165] #1: ff1100016f920280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1358.967126] #2: ff11000123e44b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1358.978273] #3: ff11000123e89d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1358.988370] 4 locks held by python3/5815: [ 1358.993667] #0: ff110001297a43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1359.003555] #1: ff110001736b2c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1359.014515] #2: ff11000123e44b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1359.025673] #3: ff11000123e89d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1359.035827] 1 lock held by systemd/7660: [ 1359.041025] #0: ff1100180e2372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0 [ 1359.050916] 2 locks held by 30-systemd-envi/7664: [ 1359.056994] [ 1359.059452] =============================================\x0a