[ 787.432498] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 787.434905] nipa-hw-worker: [5/46] Retrying drivers/net:napi_threaded.py
[ 787.585465] kselftest: Running tests in drivers/net
[ 989.711275] INFO: task python3:5814 blocked for more than 122 seconds.
[ 989.719506] Not tainted 7.1.0-rc3-jrcq-g5d620799194c #1
[ 989.727443] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.737032] task:python3 state:D stack:25672 pid:5814 tgid:5814 ppid:5813 task_flags:0x400100 flags:0x00080000
[ 989.750229] Call Trace:
[ 989.753769]
[ 989.756918] __schedule+0xcc3/0x19c0
[ 989.761733] ? io_schedule_timeout+0x140/0x140
[ 989.767506] ? find_held_lock+0x2b/0x80
[ 989.772610] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.778496] schedule+0xd8/0x260
[ 989.782912] schedule_preempt_disabled+0x18/0x30
[ 989.788889] __mutex_lock+0x6af/0x2010
[ 989.793891] ? threaded_store+0x132/0x220
[ 989.799191] ? ww_mutex_lock+0x160/0x160
[ 989.804381] ? selinux_inode_free_security+0x330/0x330
[ 989.810949] ? cap_capable+0x1d6/0x3e0
[ 989.815958] ? threaded_store+0x132/0x220
[ 989.821252] threaded_store+0x132/0x220
[ 989.826355] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.831653] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.836953] ? sysfs_kf_write+0x65/0x160
[ 989.842152] ? sysfs_file_ops+0x140/0x140
[ 989.847446] kernfs_fop_write_iter+0x335/0x510
[ 989.853230] ? kernfs_ops+0xf0/0xf0
[ 989.857943] new_sync_write+0x333/0x750
[ 989.863043] ? __lock_acquire+0x508/0xc10
[ 989.868340] ? new_sync_read+0x740/0x740
[ 989.873538] ? lock_acquire.part.0+0xbc/0x260
[ 989.879226] ? ksys_write+0x116/0x250
[ 989.884127] vfs_write+0x657/0x9b0
[ 989.888740] ksys_write+0x116/0x250
[ 989.893450] ? __ia32_sys_read+0xc0/0xc0
[ 989.898649] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.904236] ? rcu_is_watching+0x15/0xd0
[ 989.909439] do_syscall_64+0xf3/0x650
[ 989.914346] ? trace_hardirqs_off+0xd/0x30
[ 989.919740] ? exc_page_fault+0xda/0xf0
[ 989.924840] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.931307] RIP: 0033:0x7f228f5f722e
[ 989.936111] RSP: 002b:00007ffd7353fbc0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.945407] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f228f5f722e
[ 989.954215] RDX: 0000000000000001 RSI: 00005587d2439a90 RDI: 0000000000000007
[ 989.963020] RBP: 00007ffd7353fbd0 R08: 0000000000000000 R09: 0000000000000000
[ 989.971827] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f228f46f390
[ 989.980634] R13: 00005587d2439a90 R14: 0000000000000007 R15: 0000000000000003
[ 989.989445]
[ 989.992731] INFO: task python3:5814 is blocked on a mutex likely owned by task python3:3754.
[ 990.003000] task:python3 state:D stack:25672 pid:3754 tgid:3754 ppid:3753 task_flags:0x400100 flags:0x00080802
[ 990.016201] Call Trace:
[ 990.019740]
[ 990.022890] __schedule+0xcc3/0x19c0
[ 990.027699] ? io_schedule_timeout+0x140/0x140
[ 990.033480] ? find_held_lock+0x2b/0x80
[ 990.038582] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.044753] schedule+0xd8/0x260
[ 990.049168] schedule_timeout+0x13f/0x290
[ 990.054464] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.060830] ? trace_contention_end+0xb3/0x1a0
[ 990.066615] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.072304] ? ww_mutex_lock+0x160/0x160
[ 990.077503] msleep+0x1b/0x30
[ 990.081618] napi_set_threaded+0x3a5/0x440
[ 990.087015] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.092502] netif_set_threaded+0x257/0x400
[ 990.097993] threaded_store+0x1b8/0x220
[ 990.103095] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.108384] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.113677] ? sysfs_kf_write+0x65/0x160
[ 990.118875] ? sysfs_file_ops+0x140/0x140
[ 990.124169] kernfs_fop_write_iter+0x335/0x510
[ 990.129955] ? kernfs_ops+0xf0/0xf0
[ 990.134664] new_sync_write+0x333/0x750
[ 990.139765] ? __lock_acquire+0x508/0xc10
[ 990.145058] ? new_sync_read+0x740/0x740
[ 990.150257] ? lock_acquire.part.0+0xbc/0x260
[ 990.155943] ? ksys_write+0x116/0x250
[ 990.160850] vfs_write+0x657/0x9b0
[ 990.165465] ksys_write+0x116/0x250
[ 990.170168] ? __ia32_sys_read+0xc0/0xc0
[ 990.175361] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.180951] ? rcu_is_watching+0x15/0xd0
[ 990.186139] do_syscall_64+0xf3/0x650
[ 990.191041] ? irq_exit_rcu+0x1a/0x30
[ 990.195948] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.202414] RIP: 0033:0x7f22b2ac322e
[ 990.207222] RSP: 002b:00007ffcdf9dd430 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.216519] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f22b2ac322e
[ 990.225323] RDX: 0000000000000001 RSI: 000056008448aa90 RDI: 0000000000000007
[ 990.234132] RBP: 00007ffcdf9dd440 R08: 0000000000000000 R09: 0000000000000000
[ 990.242937] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f22b293b390
[ 990.251745] R13: 000056008448aa90 R14: 0000000000000007 R15: 0000000000000003
[ 990.260549]
[ 990.263807] \x0aShowing all locks held in the system:
[ 990.272346] 1 lock held by khungtaskd/185:
[ 990.277738] #0: ffffffffb3cfdf60 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.289219] 4 locks held by python3/3754:
[ 990.294516] #0: ff110001307b43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.304402] #1: ff1100014ca14480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.315364] #2: ff110001122a6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.326521] #3: ff11000112309d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.336610] 4 locks held by python3/5814:
[ 990.341906] #0: ff110001307b43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.351794] #1: ff1100022f39f480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.362746] #2: ff110001122a6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.373903] #3: ff11000112309d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.383985]
[ 990.386452] =============================================\x0a
[ 1112.591440] INFO: task python3:5814 blocked for more than 245 seconds.
[ 1112.599592] Not tainted 7.1.0-rc3-jrcq-g5d620799194c #1
[ 1112.606840] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.616429] task:python3 state:D stack:25672 pid:5814 tgid:5814 ppid:5813 task_flags:0x400100 flags:0x00080000
[ 1112.629626] Call Trace:
[ 1112.633167]
[ 1112.636315] __schedule+0xcc3/0x19c0
[ 1112.641132] ? io_schedule_timeout+0x140/0x140
[ 1112.646919] ? find_held_lock+0x2b/0x80
[ 1112.652021] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.657905] schedule+0xd8/0x260
[ 1112.662324] schedule_preempt_disabled+0x18/0x30
[ 1112.668302] __mutex_lock+0x6af/0x2010
[ 1112.673306] ? threaded_store+0x132/0x220
[ 1112.678605] ? ww_mutex_lock+0x160/0x160
[ 1112.683804] ? selinux_inode_free_security+0x330/0x330
[ 1112.690375] ? cap_capable+0x1d6/0x3e0
[ 1112.695382] ? threaded_store+0x132/0x220
[ 1112.700676] threaded_store+0x132/0x220
[ 1112.705781] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.711080] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.716377] ? sysfs_kf_write+0x65/0x160
[ 1112.721578] ? sysfs_file_ops+0x140/0x140
[ 1112.726875] kernfs_fop_write_iter+0x335/0x510
[ 1112.732658] ? kernfs_ops+0xf0/0xf0
[ 1112.737370] new_sync_write+0x333/0x750
[ 1112.742471] ? __lock_acquire+0x508/0xc10
[ 1112.747767] ? new_sync_read+0x740/0x740
[ 1112.752969] ? lock_acquire.part.0+0xbc/0x260
[ 1112.758651] ? ksys_write+0x116/0x250
[ 1112.763561] vfs_write+0x657/0x9b0
[ 1112.768177] ksys_write+0x116/0x250
[ 1112.772889] ? __ia32_sys_read+0xc0/0xc0
[ 1112.778086] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.783672] ? rcu_is_watching+0x15/0xd0
[ 1112.788874] do_syscall_64+0xf3/0x650
[ 1112.793778] ? trace_hardirqs_off+0xd/0x30
[ 1112.799170] ? exc_page_fault+0xda/0xf0
[ 1112.804275] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.810741] RIP: 0033:0x7f228f5f722e
[ 1112.815548] RSP: 002b:00007ffd7353fbc0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.824846] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f228f5f722e
[ 1112.833652] RDX: 0000000000000001 RSI: 00005587d2439a90 RDI: 0000000000000007
[ 1112.842456] RBP: 00007ffd7353fbd0 R08: 0000000000000000 R09: 0000000000000000
[ 1112.851340] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f228f46f390
[ 1112.860148] R13: 00005587d2439a90 R14: 0000000000000007 R15: 0000000000000003
[ 1112.868962]
[ 1112.872290] INFO: task python3:5814 is blocked on a mutex likely owned by task python3:3754.
[ 1112.882560] task:python3 state:D stack:25672 pid:3754 tgid:3754 ppid:3753 task_flags:0x400100 flags:0x00080802
[ 1112.895760] Call Trace:
[ 1112.899296]
[ 1112.902445] __schedule+0xcc3/0x19c0
[ 1112.907255] ? io_schedule_timeout+0x140/0x140
[ 1112.913030] ? find_held_lock+0x2b/0x80
[ 1112.918132] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.924298] schedule+0xd8/0x260
[ 1112.928714] schedule_timeout+0x13f/0x290
[ 1112.934009] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.940378] ? trace_contention_end+0xb3/0x1a0
[ 1112.946163] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.951845] ? ww_mutex_lock+0x160/0x160
[ 1112.957043] msleep+0x1b/0x30
[ 1112.961167] napi_set_threaded+0x3a5/0x440
[ 1112.966564] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.972056] netif_set_threaded+0x257/0x400
[ 1112.977548] threaded_store+0x1b8/0x220
[ 1112.982641] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.987939] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.993233] ? sysfs_kf_write+0x65/0x160
[ 1112.998430] ? sysfs_file_ops+0x140/0x140
[ 1113.003726] kernfs_fop_write_iter+0x335/0x510
[ 1113.009508] ? kernfs_ops+0xf0/0xf0
[ 1113.014219] new_sync_write+0x333/0x750
[ 1113.019320] ? __lock_acquire+0x508/0xc10
[ 1113.024615] ? new_sync_read+0x740/0x740
[ 1113.029816] ? lock_acquire.part.0+0xbc/0x260
[ 1113.035492] ? ksys_write+0x116/0x250
[ 1113.040400] vfs_write+0x657/0x9b0
[ 1113.045016] ksys_write+0x116/0x250
[ 1113.049729] ? __ia32_sys_read+0xc0/0xc0
[ 1113.054920] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.060498] ? rcu_is_watching+0x15/0xd0
[ 1113.065695] do_syscall_64+0xf3/0x650
[ 1113.070603] ? irq_exit_rcu+0x1a/0x30
[ 1113.075573] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.082040] RIP: 0033:0x7f22b2ac322e
[ 1113.086847] RSP: 002b:00007ffcdf9dd430 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.096144] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f22b2ac322e
[ 1113.105016] RDX: 0000000000000001 RSI: 000056008448aa90 RDI: 0000000000000007
[ 1113.113823] RBP: 00007ffcdf9dd440 R08: 0000000000000000 R09: 0000000000000000
[ 1113.122631] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f22b293b390
[ 1113.131429] R13: 000056008448aa90 R14: 0000000000000007 R15: 0000000000000003
[ 1113.140239]
[ 1113.143566] \x0aShowing all locks held in the system:
[ 1113.152114] 1 lock held by khungtaskd/185:
[ 1113.157507] #0: ffffffffb3cfdf60 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.168982] 3 locks held by systemd-journal/802:
[ 1113.174961] #0: ff1100180dfb72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1113.184851] #1: ffffffffb3f02b20 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: nd_jump_root+0x353/0x550
[ 1113.197177] #2: ff1100019d5018e0 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 1113.208735] 4 locks held by python3/3754:
[ 1113.214030] #0: ff110001307b43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.223918] #1: ff1100014ca14480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.234877] #2: ff110001122a6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.246095] #3: ff11000112309d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.256200] 4 locks held by python3/5814:
[ 1113.261498] #0: ff110001307b43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.271386] #1: ff1100022f39f480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.282350] #2: ff110001122a6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.293507] #3: ff11000112309d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.303601] 2 locks held by systemd/6862:
[ 1113.308914]
[ 1113.311383] =============================================\x0a
[ 1235.472581] INFO: task python3:5814 blocked for more than 368 seconds.
[ 1235.480727] Not tainted 7.1.0-rc3-jrcq-g5d620799194c #1
[ 1235.487976] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.497566] task:python3 state:D stack:25672 pid:5814 tgid:5814 ppid:5813 task_flags:0x400100 flags:0x00080002
[ 1235.510770] Call Trace:
[ 1235.514306]
[ 1235.517455] __schedule+0xcc3/0x19c0
[ 1235.522271] ? io_schedule_timeout+0x140/0x140
[ 1235.528054] ? find_held_lock+0x2b/0x80
[ 1235.533158] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.539041] schedule+0xd8/0x260
[ 1235.543456] schedule_preempt_disabled+0x18/0x30
[ 1235.549432] __mutex_lock+0x6af/0x2010
[ 1235.554434] ? threaded_store+0x132/0x220
[ 1235.559732] ? ww_mutex_lock+0x160/0x160
[ 1235.564929] ? selinux_inode_free_security+0x330/0x330
[ 1235.571489] ? cap_capable+0x1d6/0x3e0
[ 1235.576498] ? threaded_store+0x132/0x220
[ 1235.581794] threaded_store+0x132/0x220
[ 1235.586894] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.592191] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.597489] ? sysfs_kf_write+0x65/0x160
[ 1235.602692] ? sysfs_file_ops+0x140/0x140
[ 1235.607987] kernfs_fop_write_iter+0x335/0x510
[ 1235.613769] ? kernfs_ops+0xf0/0xf0
[ 1235.618479] new_sync_write+0x333/0x750
[ 1235.623573] ? __lock_acquire+0x508/0xc10
[ 1235.628867] ? new_sync_read+0x740/0x740
[ 1235.634068] ? lock_acquire.part.0+0xbc/0x260
[ 1235.639754] ? ksys_write+0x116/0x250
[ 1235.644663] vfs_write+0x657/0x9b0
[ 1235.649278] ksys_write+0x116/0x250
[ 1235.653980] ? __ia32_sys_read+0xc0/0xc0
[ 1235.659177] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.664763] ? rcu_is_watching+0x15/0xd0
[ 1235.669962] do_syscall_64+0xf3/0x650
[ 1235.674869] ? trace_hardirqs_off+0xd/0x30
[ 1235.680263] ? exc_page_fault+0xda/0xf0
[ 1235.685365] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.691833] RIP: 0033:0x7f228f5f722e
[ 1235.696643] RSP: 002b:00007ffd7353fbc0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.705933] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f228f5f722e
[ 1235.714739] RDX: 0000000000000001 RSI: 00005587d2439a90 RDI: 0000000000000007
[ 1235.723544] RBP: 00007ffd7353fbd0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.732350] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f228f46f390
[ 1235.741160] R13: 00005587d2439a90 R14: 0000000000000007 R15: 0000000000000003
[ 1235.749973]
[ 1235.753259] INFO: task python3:5814 is blocked on a mutex likely owned by task python3:3754.
[ 1235.763529] task:python3 state:D stack:25672 pid:3754 tgid:3754 ppid:3753 task_flags:0x400100 flags:0x00080802
[ 1235.776732] Call Trace:
[ 1235.780268]
[ 1235.783421] __schedule+0xcc3/0x19c0
[ 1235.788232] ? io_schedule_timeout+0x140/0x140
[ 1235.794017] ? find_held_lock+0x2b/0x80
[ 1235.799111] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.805288] schedule+0xd8/0x260
[ 1235.809705] schedule_timeout+0x13f/0x290
[ 1235.815001] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.821371] ? trace_contention_end+0xb3/0x1a0
[ 1235.827148] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.832836] ? ww_mutex_lock+0x160/0x160
[ 1235.838035] msleep+0x1b/0x30
[ 1235.842160] napi_set_threaded+0x3a5/0x440
[ 1235.847557] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.853051] netif_set_threaded+0x257/0x400
[ 1235.858535] threaded_store+0x1b8/0x220
[ 1235.863635] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.868931] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.874225] ? sysfs_kf_write+0x65/0x160
[ 1235.879427] ? sysfs_file_ops+0x140/0x140
[ 1235.884720] kernfs_fop_write_iter+0x335/0x510
[ 1235.890504] ? kernfs_ops+0xf0/0xf0
[ 1235.895215] new_sync_write+0x333/0x750
[ 1235.900315] ? __lock_acquire+0x508/0xc10
[ 1235.905609] ? new_sync_read+0x740/0x740
[ 1235.910806] ? lock_acquire.part.0+0xbc/0x260
[ 1235.916490] ? ksys_write+0x116/0x250
[ 1235.921400] vfs_write+0x657/0x9b0
[ 1235.926015] ksys_write+0x116/0x250
[ 1235.930717] ? __ia32_sys_read+0xc0/0xc0
[ 1235.935916] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.941504] ? rcu_is_watching+0x15/0xd0
[ 1235.946705] do_syscall_64+0xf3/0x650
[ 1235.951600] ? irq_exit_rcu+0x1a/0x30
[ 1235.956505] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.962970] RIP: 0033:0x7f22b2ac322e
[ 1235.967776] RSP: 002b:00007ffcdf9dd430 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.977072] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f22b2ac322e
[ 1235.985878] RDX: 0000000000000001 RSI: 000056008448aa90 RDI: 0000000000000007
[ 1235.994684] RBP: 00007ffcdf9dd440 R08: 0000000000000000 R09: 0000000000000000
[ 1236.003491] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f22b293b390
[ 1236.012299] R13: 000056008448aa90 R14: 0000000000000007 R15: 0000000000000003
[ 1236.021109]
[ 1236.024374] \x0aShowing all locks held in the system:
[ 1236.032910] 1 lock held by khungtaskd/185:
[ 1236.038301] #0: ffffffffb3cfdf60 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.049772] 3 locks held by systemd-journal/802:
[ 1236.055748] #0: ff1100180deb72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1236.065632] #1: ffffffffb3f02b20 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1236.078445] #2: ff1100019d5018e0 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 1236.090003] 4 locks held by python3/3754:
[ 1236.095299] #0: ff110001307b43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.105185] #1: ff1100014ca14480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.116147] #2: ff110001122a6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.127301] #3: ff11000112309d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.137399] 4 locks held by python3/5814:
[ 1236.142692] #0: ff110001307b43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.152578] #1: ff1100022f39f480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.163537] #2: ff110001122a6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.174692] #3: ff11000112309d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.184781]
[ 1236.187247] =============================================\x0a
[ 1358.354731] INFO: task python3:5814 blocked for more than 491 seconds.
[ 1358.362889] Not tainted 7.1.0-rc3-jrcq-g5d620799194c #1
[ 1358.370140] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.379731] task:python3 state:D stack:25672 pid:5814 tgid:5814 ppid:5813 task_flags:0x400100 flags:0x00080002
[ 1358.392936] Call Trace:
[ 1358.396474]
[ 1358.399625] __schedule+0xcc3/0x19c0
[ 1358.404440] ? io_schedule_timeout+0x140/0x140
[ 1358.410225] ? find_held_lock+0x2b/0x80
[ 1358.415328] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.421212] schedule+0xd8/0x260
[ 1358.425631] schedule_preempt_disabled+0x18/0x30
[ 1358.431609] __mutex_lock+0x6af/0x2010
[ 1358.436617] ? threaded_store+0x132/0x220
[ 1358.441914] ? ww_mutex_lock+0x160/0x160
[ 1358.447111] ? selinux_inode_free_security+0x330/0x330
[ 1358.453679] ? cap_capable+0x1d6/0x3e0
[ 1358.458684] ? threaded_store+0x132/0x220
[ 1358.463970] threaded_store+0x132/0x220
[ 1358.469072] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.474367] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.479663] ? sysfs_kf_write+0x65/0x160
[ 1358.484862] ? sysfs_file_ops+0x140/0x140
[ 1358.490158] kernfs_fop_write_iter+0x335/0x510
[ 1358.495941] ? kernfs_ops+0xf0/0xf0
[ 1358.500651] new_sync_write+0x333/0x750
[ 1358.505755] ? __lock_acquire+0x508/0xc10
[ 1358.511049] ? new_sync_read+0x740/0x740
[ 1358.516250] ? lock_acquire.part.0+0xbc/0x260
[ 1358.521934] ? ksys_write+0x116/0x250
[ 1358.526844] vfs_write+0x657/0x9b0
[ 1358.531457] ksys_write+0x116/0x250
[ 1358.536166] ? __ia32_sys_read+0xc0/0xc0
[ 1358.541433] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.547022] ? rcu_is_watching+0x15/0xd0
[ 1358.552223] do_syscall_64+0xf3/0x650
[ 1358.557131] ? trace_hardirqs_off+0xd/0x30
[ 1358.562525] ? exc_page_fault+0xda/0xf0
[ 1358.567628] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.574097] RIP: 0033:0x7f228f5f722e
[ 1358.578907] RSP: 002b:00007ffd7353fbc0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.588203] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f228f5f722e
[ 1358.597011] RDX: 0000000000000001 RSI: 00005587d2439a90 RDI: 0000000000000007
[ 1358.605819] RBP: 00007ffd7353fbd0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.614617] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f228f46f390
[ 1358.623426] R13: 00005587d2439a90 R14: 0000000000000007 R15: 0000000000000003
[ 1358.632240]
[ 1358.635554] INFO: task python3:5814 is blocked on a mutex likely owned by task python3:3754.
[ 1358.645830] task:python3 state:D stack:25672 pid:3754 tgid:3754 ppid:3753 task_flags:0x400100 flags:0x00080802
[ 1358.659031] Call Trace:
[ 1358.662569]
[ 1358.665721] __schedule+0xcc3/0x19c0
[ 1358.670531] ? io_schedule_timeout+0x140/0x140
[ 1358.676315] ? find_held_lock+0x2b/0x80
[ 1358.681408] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.687586] schedule+0xd8/0x260
[ 1358.692004] schedule_timeout+0x13f/0x290
[ 1358.697290] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.703657] ? trace_contention_end+0xb3/0x1a0
[ 1358.709440] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.715129] ? ww_mutex_lock+0x160/0x160
[ 1358.720326] msleep+0x1b/0x30
[ 1358.724453] napi_set_threaded+0x3a5/0x440
[ 1358.729848] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.735342] netif_set_threaded+0x257/0x400
[ 1358.740836] threaded_store+0x1b8/0x220
[ 1358.745939] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.751227] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.756523] ? sysfs_kf_write+0x65/0x160
[ 1358.761722] ? sysfs_file_ops+0x140/0x140
[ 1358.767014] kernfs_fop_write_iter+0x335/0x510
[ 1358.772800] ? kernfs_ops+0xf0/0xf0
[ 1358.777510] new_sync_write+0x333/0x750
[ 1358.782612] ? __lock_acquire+0x508/0xc10
[ 1358.787908] ? new_sync_read+0x740/0x740
[ 1358.793106] ? lock_acquire.part.0+0xbc/0x260
[ 1358.798792] ? ksys_write+0x116/0x250
[ 1358.803701] vfs_write+0x657/0x9b0
[ 1358.808313] ksys_write+0x116/0x250
[ 1358.813025] ? __ia32_sys_read+0xc0/0xc0
[ 1358.818225] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.823804] ? rcu_is_watching+0x15/0xd0
[ 1358.829002] do_syscall_64+0xf3/0x650
[ 1358.833909] ? irq_exit_rcu+0x1a/0x30
[ 1358.838814] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.845279] RIP: 0033:0x7f22b2ac322e
[ 1358.850086] RSP: 002b:00007ffcdf9dd430 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.859384] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f22b2ac322e
[ 1358.868191] RDX: 0000000000000001 RSI: 000056008448aa90 RDI: 0000000000000007
[ 1358.877000] RBP: 00007ffcdf9dd440 R08: 0000000000000000 R09: 0000000000000000
[ 1358.885806] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f22b293b390
[ 1358.894612] R13: 000056008448aa90 R14: 0000000000000007 R15: 0000000000000003
[ 1358.903424]
[ 1358.906694] \x0aShowing all locks held in the system:
[ 1358.915237] 1 lock held by khungtaskd/185:
[ 1358.920632] #0: ffffffffb3cfdf60 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.932120] 4 locks held by python3/3754:
[ 1358.937414] #0: ff110001307b43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.947301] #1: ff1100014ca14480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.958260] #2: ff110001122a6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.969414] #3: ff11000112309d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.979509] 4 locks held by python3/5814:
[ 1358.984804] #0: ff110001307b43e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.994691] #1: ff1100022f39f480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1359.005650] #2: ff110001122a6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.016804] #3: ff11000112309d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.026895] 1 lock held by systemd/7657:
[ 1359.032093]
[ 1359.034565] =============================================\x0a