[ 790.092393] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 790.094867] nipa-hw-worker: [5/46] Retrying drivers/net:napi_threaded.py
[ 790.227162] kselftest: Running tests in drivers/net
[ 838.140702] kworker/u129:14 (311) used greatest stack depth: 21456 bytes left
[ 989.695518] INFO: task python3:5851 blocked for more than 122 seconds.
[ 989.703742] Not tainted 7.1.0-rc5-qjtz-g9b5731360ceb #1
[ 989.711494] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.721084] task:python3 state:D stack:25672 pid:5851 tgid:5851 ppid:5850 task_flags:0x400100 flags:0x00080000
[ 989.734289] Call Trace:
[ 989.737828]
[ 989.740976] __schedule+0xcc3/0x19c0
[ 989.745794] ? io_schedule_timeout+0x140/0x140
[ 989.751580] ? find_held_lock+0x2b/0x80
[ 989.756682] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.762566] schedule+0xd8/0x260
[ 989.766982] schedule_preempt_disabled+0x18/0x30
[ 989.772961] __mutex_lock+0x6af/0x2010
[ 989.777967] ? threaded_store+0x132/0x220
[ 989.783268] ? ww_mutex_lock+0x160/0x160
[ 989.788465] ? selinux_inode_free_security+0x330/0x330
[ 989.795033] ? cap_capable+0x1d6/0x3e0
[ 989.800041] ? threaded_store+0x132/0x220
[ 989.805338] threaded_store+0x132/0x220
[ 989.810441] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.815738] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.821037] ? sysfs_kf_write+0x65/0x160
[ 989.826236] ? sysfs_file_ops+0x140/0x140
[ 989.831531] kernfs_fop_write_iter+0x335/0x510
[ 989.837317] ? kernfs_ops+0xf0/0xf0
[ 989.842027] new_sync_write+0x333/0x750
[ 989.847131] ? __lock_acquire+0x508/0xc10
[ 989.852425] ? new_sync_read+0x740/0x740
[ 989.857625] ? lock_acquire.part.0+0xbc/0x260
[ 989.863302] ? ksys_write+0x116/0x250
[ 989.868210] vfs_write+0x657/0x9b0
[ 989.872826] ksys_write+0x116/0x250
[ 989.877538] ? __ia32_sys_read+0xc0/0xc0
[ 989.882726] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.888315] ? rcu_is_watching+0x15/0xd0
[ 989.893516] do_syscall_64+0xf3/0x650
[ 989.898420] ? trace_hardirqs_off+0xd/0x30
[ 989.903815] ? exc_page_fault+0xda/0xf0
[ 989.908917] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.915387] RIP: 0033:0x7f012e60e22e
[ 989.920197] RSP: 002b:00007ffca7aad300 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.929494] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f012e60e22e
[ 989.938302] RDX: 0000000000000001 RSI: 000055dc59c286b0 RDI: 0000000000000007
[ 989.947101] RBP: 00007ffca7aad310 R08: 0000000000000000 R09: 0000000000000000
[ 989.955901] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f012e486390
[ 989.964709] R13: 000055dc59c286b0 R14: 0000000000000007 R15: 0000000000000003
[ 989.973524]
[ 989.976812] INFO: task python3:5851 is blocked on a mutex likely owned by task python3:3779.
[ 989.987081] task:python3 state:D stack:25672 pid:3779 tgid:3779 ppid:3778 task_flags:0x400100 flags:0x00080802
[ 990.000282] Call Trace:
[ 990.003819]
[ 990.006967] __schedule+0xcc3/0x19c0
[ 990.011778] ? io_schedule_timeout+0x140/0x140
[ 990.017562] ? find_held_lock+0x2b/0x80
[ 990.022663] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.028839] schedule+0xd8/0x260
[ 990.033257] schedule_timeout+0x13f/0x290
[ 990.038554] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.044923] ? trace_contention_end+0xb3/0x1a0
[ 990.050706] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.056394] ? ww_mutex_lock+0x160/0x160
[ 990.061586] msleep+0x1b/0x30
[ 990.065712] napi_set_threaded+0x3a5/0x440
[ 990.071104] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.076597] netif_set_threaded+0x257/0x400
[ 990.082086] threaded_store+0x1b8/0x220
[ 990.087185] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.092483] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.097778] ? sysfs_kf_write+0x65/0x160
[ 990.102978] ? sysfs_file_ops+0x140/0x140
[ 990.108274] kernfs_fop_write_iter+0x335/0x510
[ 990.114057] ? kernfs_ops+0xf0/0xf0
[ 990.118768] new_sync_write+0x333/0x750
[ 990.123871] ? __lock_acquire+0x508/0xc10
[ 990.129156] ? new_sync_read+0x740/0x740
[ 990.134358] ? lock_acquire.part.0+0xbc/0x260
[ 990.140044] ? ksys_write+0x116/0x250
[ 990.144953] vfs_write+0x657/0x9b0
[ 990.149565] ksys_write+0x116/0x250
[ 990.154274] ? __ia32_sys_read+0xc0/0xc0
[ 990.159472] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.165059] ? rcu_is_watching+0x15/0xd0
[ 990.170258] do_syscall_64+0xf3/0x650
[ 990.175162] ? trace_hardirqs_off+0xd/0x30
[ 990.180557] ? exc_page_fault+0xda/0xf0
[ 990.185650] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.192115] RIP: 0033:0x7fb564c8d22e
[ 990.196920] RSP: 002b:00007ffea3ecb0d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.206218] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb564c8d22e
[ 990.215022] RDX: 0000000000000001 RSI: 0000561e58d266b0 RDI: 0000000000000007
[ 990.223832] RBP: 00007ffea3ecb0e0 R08: 0000000000000000 R09: 0000000000000000
[ 990.232637] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fb564b05390
[ 990.241441] R13: 0000561e58d266b0 R14: 0000000000000007 R15: 0000000000000003
[ 990.250255]
[ 990.253514] \x0aShowing all locks held in the system:
[ 990.262050] 1 lock held by khungtaskd/184:
[ 990.267442] #0: ffffffffb88fdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.278917] 2 locks held by systemd-journal/801:
[ 990.284892] #0: ff1100180e8b72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 990.294779] #1: ffffffffb8b02c20 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: dput.part.0+0x2fd/0x410
[ 990.307021] 4 locks held by python3/3779:
[ 990.312316] #0: ff1100012fe993e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.322206] #1: ff11000188bfb880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.333166] #2: ff11000122d70b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.344320] #3: ff11000122e11d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.354419] 4 locks held by python3/5851:
[ 990.359716] #0: ff1100012fe993e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.369604] #1: ff11000107138280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.380562] #2: ff11000122d70b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.391717] #3: ff11000122e11d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.401811]
[ 990.404286] =============================================\x0a
[ 1112.575947] INFO: task python3:5851 blocked for more than 245 seconds.
[ 1112.584095] Not tainted 7.1.0-rc5-qjtz-g9b5731360ceb #1
[ 1112.591336] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.600928] task:python3 state:D stack:25672 pid:5851 tgid:5851 ppid:5850 task_flags:0x400100 flags:0x00080000
[ 1112.614135] Call Trace:
[ 1112.617675]
[ 1112.620827] __schedule+0xcc3/0x19c0
[ 1112.625645] ? io_schedule_timeout+0x140/0x140
[ 1112.631431] ? find_held_lock+0x2b/0x80
[ 1112.636536] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.642411] schedule+0xd8/0x260
[ 1112.646832] schedule_preempt_disabled+0x18/0x30
[ 1112.652813] __mutex_lock+0x6af/0x2010
[ 1112.657821] ? threaded_store+0x132/0x220
[ 1112.663118] ? ww_mutex_lock+0x160/0x160
[ 1112.668319] ? selinux_inode_free_security+0x330/0x330
[ 1112.674887] ? cap_capable+0x1d6/0x3e0
[ 1112.679896] ? threaded_store+0x132/0x220
[ 1112.685193] threaded_store+0x132/0x220
[ 1112.690294] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.695592] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.700896] ? sysfs_kf_write+0x65/0x160
[ 1112.706097] ? sysfs_file_ops+0x140/0x140
[ 1112.711394] kernfs_fop_write_iter+0x335/0x510
[ 1112.717181] ? kernfs_ops+0xf0/0xf0
[ 1112.721893] new_sync_write+0x333/0x750
[ 1112.726996] ? __lock_acquire+0x508/0xc10
[ 1112.732295] ? new_sync_read+0x740/0x740
[ 1112.737497] ? lock_acquire.part.0+0xbc/0x260
[ 1112.743182] ? ksys_write+0x116/0x250
[ 1112.748094] vfs_write+0x657/0x9b0
[ 1112.752711] ksys_write+0x116/0x250
[ 1112.757424] ? __ia32_sys_read+0xc0/0xc0
[ 1112.762625] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.768217] ? rcu_is_watching+0x15/0xd0
[ 1112.773421] do_syscall_64+0xf3/0x650
[ 1112.778330] ? trace_hardirqs_off+0xd/0x30
[ 1112.783725] ? exc_page_fault+0xda/0xf0
[ 1112.788827] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.795295] RIP: 0033:0x7f012e60e22e
[ 1112.800105] RSP: 002b:00007ffca7aad300 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.809393] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f012e60e22e
[ 1112.818193] RDX: 0000000000000001 RSI: 000055dc59c286b0 RDI: 0000000000000007
[ 1112.826991] RBP: 00007ffca7aad310 R08: 0000000000000000 R09: 0000000000000000
[ 1112.835801] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f012e486390
[ 1112.844610] R13: 000055dc59c286b0 R14: 0000000000000007 R15: 0000000000000003
[ 1112.853423]
[ 1112.856706] INFO: task python3:5851 is blocked on a mutex likely owned by task python3:3779.
[ 1112.866971] task:python3 state:D stack:25672 pid:3779 tgid:3779 ppid:3778 task_flags:0x400100 flags:0x00080802
[ 1112.880174] Call Trace:
[ 1112.883713]
[ 1112.886864] __schedule+0xcc3/0x19c0
[ 1112.891677] ? io_schedule_timeout+0x140/0x140
[ 1112.897462] ? find_held_lock+0x2b/0x80
[ 1112.902567] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.908743] schedule+0xd8/0x260
[ 1112.913162] schedule_timeout+0x13f/0x290
[ 1112.918459] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.924828] ? trace_contention_end+0xb3/0x1a0
[ 1112.930611] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.936303] ? ww_mutex_lock+0x160/0x160
[ 1112.941503] msleep+0x1b/0x30
[ 1112.945628] napi_set_threaded+0x3a5/0x440
[ 1112.951026] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.956519] netif_set_threaded+0x257/0x400
[ 1112.962014] threaded_store+0x1b8/0x220
[ 1112.967115] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.972414] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.977713] ? sysfs_kf_write+0x65/0x160
[ 1112.982913] ? sysfs_file_ops+0x140/0x140
[ 1112.988201] kernfs_fop_write_iter+0x335/0x510
[ 1112.993988] ? kernfs_ops+0xf0/0xf0
[ 1112.998699] new_sync_write+0x333/0x750
[ 1113.003804] ? __lock_acquire+0x508/0xc10
[ 1113.009102] ? new_sync_read+0x740/0x740
[ 1113.014303] ? lock_acquire.part.0+0xbc/0x260
[ 1113.019988] ? ksys_write+0x116/0x250
[ 1113.024902] vfs_write+0x657/0x9b0
[ 1113.029518] ksys_write+0x116/0x250
[ 1113.034231] ? __ia32_sys_read+0xc0/0xc0
[ 1113.039433] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.045023] ? rcu_is_watching+0x15/0xd0
[ 1113.050212] do_syscall_64+0xf3/0x650
[ 1113.055121] ? trace_hardirqs_off+0xd/0x30
[ 1113.060515] ? exc_page_fault+0xda/0xf0
[ 1113.065617] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.072086] RIP: 0033:0x7fb564c8d22e
[ 1113.076895] RSP: 002b:00007ffea3ecb0d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.086182] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb564c8d22e
[ 1113.094992] RDX: 0000000000000001 RSI: 0000561e58d266b0 RDI: 0000000000000007
[ 1113.103803] RBP: 00007ffea3ecb0e0 R08: 0000000000000000 R09: 0000000000000000
[ 1113.112612] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fb564b05390
[ 1113.121420] R13: 0000561e58d266b0 R14: 0000000000000007 R15: 0000000000000003
[ 1113.130224]
[ 1113.133486] \x0aShowing all locks held in the system:
[ 1113.142024] 1 lock held by khungtaskd/184:
[ 1113.147416] #0: ffffffffb88fdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.158900] 4 locks held by python3/3779:
[ 1113.164187] #0: ff1100012fe993e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.174080] #1: ff11000188bfb880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.185040] #2: ff11000122d70b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.196195] #3: ff11000122e11d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.206291] 4 locks held by python3/5851:
[ 1113.211586] #0: ff1100012fe993e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.221476] #1: ff11000107138280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.232440] #2: ff11000122d70b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.243602] #3: ff11000122e11d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.253693]
[ 1113.256161] =============================================\x0a
[ 1235.458463] INFO: task python3:5851 blocked for more than 368 seconds.
[ 1235.466602] Not tainted 7.1.0-rc5-qjtz-g9b5731360ceb #1
[ 1235.473843] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.483435] task:python3 state:D stack:25672 pid:5851 tgid:5851 ppid:5850 task_flags:0x400100 flags:0x00080002
[ 1235.496643] Call Trace:
[ 1235.500183]
[ 1235.503336] __schedule+0xcc3/0x19c0
[ 1235.508155] ? io_schedule_timeout+0x140/0x140
[ 1235.513941] ? find_held_lock+0x2b/0x80
[ 1235.519048] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.524933] schedule+0xd8/0x260
[ 1235.529352] schedule_preempt_disabled+0x18/0x30
[ 1235.535324] __mutex_lock+0x6af/0x2010
[ 1235.540331] ? threaded_store+0x132/0x220
[ 1235.545634] ? ww_mutex_lock+0x160/0x160
[ 1235.550834] ? selinux_inode_free_security+0x330/0x330
[ 1235.557403] ? cap_capable+0x1d6/0x3e0
[ 1235.562413] ? threaded_store+0x132/0x220
[ 1235.567712] threaded_store+0x132/0x220
[ 1235.572813] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.578111] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.583408] ? sysfs_kf_write+0x65/0x160
[ 1235.588608] ? sysfs_file_ops+0x140/0x140
[ 1235.593896] kernfs_fop_write_iter+0x335/0x510
[ 1235.599680] ? kernfs_ops+0xf0/0xf0
[ 1235.604391] new_sync_write+0x333/0x750
[ 1235.609496] ? __lock_acquire+0x508/0xc10
[ 1235.614792] ? new_sync_read+0x740/0x740
[ 1235.619993] ? lock_acquire.part.0+0xbc/0x260
[ 1235.625679] ? ksys_write+0x116/0x250
[ 1235.630589] vfs_write+0x657/0x9b0
[ 1235.635206] ksys_write+0x116/0x250
[ 1235.639919] ? __ia32_sys_read+0xc0/0xc0
[ 1235.645116] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.650706] ? rcu_is_watching+0x15/0xd0
[ 1235.655908] do_syscall_64+0xf3/0x650
[ 1235.660816] ? trace_hardirqs_off+0xd/0x30
[ 1235.666212] ? exc_page_fault+0xda/0xf0
[ 1235.671315] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.677786] RIP: 0033:0x7f012e60e22e
[ 1235.682610] RSP: 002b:00007ffca7aad300 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.691915] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f012e60e22e
[ 1235.700725] RDX: 0000000000000001 RSI: 000055dc59c286b0 RDI: 0000000000000007
[ 1235.709526] RBP: 00007ffca7aad310 R08: 0000000000000000 R09: 0000000000000000
[ 1235.718336] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f012e486390
[ 1235.727144] R13: 000055dc59c286b0 R14: 0000000000000007 R15: 0000000000000003
[ 1235.735949]
[ 1235.739264] INFO: task python3:5851 is blocked on a mutex likely owned by task python3:3779.
[ 1235.749541] task:python3 state:D stack:25672 pid:3779 tgid:3779 ppid:3778 task_flags:0x400100 flags:0x00080802
[ 1235.762749] Call Trace:
[ 1235.766286]
[ 1235.769438] __schedule+0xcc3/0x19c0
[ 1235.774252] ? io_schedule_timeout+0x140/0x140
[ 1235.780035] ? find_held_lock+0x2b/0x80
[ 1235.785140] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.791321] schedule+0xd8/0x260
[ 1235.795739] schedule_timeout+0x13f/0x290
[ 1235.801035] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.807408] ? trace_contention_end+0xb3/0x1a0
[ 1235.813194] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.818884] ? ww_mutex_lock+0x160/0x160
[ 1235.824084] msleep+0x1b/0x30
[ 1235.828199] napi_set_threaded+0x3a5/0x440
[ 1235.833595] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.839083] netif_set_threaded+0x257/0x400
[ 1235.844575] threaded_store+0x1b8/0x220
[ 1235.849680] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.854977] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.860276] ? sysfs_kf_write+0x65/0x160
[ 1235.865477] ? sysfs_file_ops+0x140/0x140
[ 1235.870772] kernfs_fop_write_iter+0x335/0x510
[ 1235.876559] ? kernfs_ops+0xf0/0xf0
[ 1235.881268] new_sync_write+0x333/0x750
[ 1235.886370] ? __lock_acquire+0x508/0xc10
[ 1235.891665] ? new_sync_read+0x740/0x740
[ 1235.896866] ? lock_acquire.part.0+0xbc/0x260
[ 1235.902541] ? ksys_write+0x116/0x250
[ 1235.907450] vfs_write+0x657/0x9b0
[ 1235.912067] ksys_write+0x116/0x250
[ 1235.916781] ? __ia32_sys_read+0xc0/0xc0
[ 1235.921978] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.927567] ? rcu_is_watching+0x15/0xd0
[ 1235.932761] do_syscall_64+0xf3/0x650
[ 1235.937669] ? trace_hardirqs_off+0xd/0x30
[ 1235.943063] ? exc_page_fault+0xda/0xf0
[ 1235.948165] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.954632] RIP: 0033:0x7fb564c8d22e
[ 1235.959442] RSP: 002b:00007ffea3ecb0d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.968742] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb564c8d22e
[ 1235.977551] RDX: 0000000000000001 RSI: 0000561e58d266b0 RDI: 0000000000000007
[ 1235.986360] RBP: 00007ffea3ecb0e0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.995174] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fb564b05390
[ 1236.003982] R13: 0000561e58d266b0 R14: 0000000000000007 R15: 0000000000000003
[ 1236.012795]
[ 1236.016059] \x0aShowing all locks held in the system:
[ 1236.024604] 1 lock held by khungtaskd/184:
[ 1236.030000] #0: ffffffffb88fdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.041491] 4 locks held by python3/3779:
[ 1236.046786] #0: ff1100012fe993e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.056680] #1: ff11000188bfb880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.067643] #2: ff11000122d70b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.078802] #3: ff11000122e11d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.088902] 4 locks held by python3/5851:
[ 1236.094196] #0: ff1100012fe993e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.104087] #1: ff11000107138280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.115049] #2: ff11000122d70b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.126208] #3: ff11000122e11d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.136303]
[ 1236.138773] =============================================\x0a
[ 1358.338962] INFO: task python3:5851 blocked for more than 491 seconds.
[ 1358.347108] Not tainted 7.1.0-rc5-qjtz-g9b5731360ceb #1
[ 1358.354354] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.363945] task:python3 state:D stack:25672 pid:5851 tgid:5851 ppid:5850 task_flags:0x400100 flags:0x00080002
[ 1358.377160] Call Trace:
[ 1358.380701]
[ 1358.383858] __schedule+0xcc3/0x19c0
[ 1358.388678] ? io_schedule_timeout+0x140/0x140
[ 1358.394468] ? find_held_lock+0x2b/0x80
[ 1358.399567] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.405459] schedule+0xd8/0x260
[ 1358.409881] schedule_preempt_disabled+0x18/0x30
[ 1358.415865] __mutex_lock+0x6af/0x2010
[ 1358.420872] ? threaded_store+0x132/0x220
[ 1358.426179] ? ww_mutex_lock+0x160/0x160
[ 1358.431374] ? selinux_inode_free_security+0x330/0x330
[ 1358.437955] ? cap_capable+0x1d6/0x3e0
[ 1358.442966] ? threaded_store+0x132/0x220
[ 1358.448273] threaded_store+0x132/0x220
[ 1358.453379] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.458679] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.463983] ? sysfs_kf_write+0x65/0x160
[ 1358.469188] ? sysfs_file_ops+0x140/0x140
[ 1358.474490] kernfs_fop_write_iter+0x335/0x510
[ 1358.480278] ? kernfs_ops+0xf0/0xf0
[ 1358.484994] new_sync_write+0x333/0x750
[ 1358.490100] ? __lock_acquire+0x508/0xc10
[ 1358.495398] ? new_sync_read+0x740/0x740
[ 1358.500600] ? lock_acquire.part.0+0xbc/0x260
[ 1358.506290] ? ksys_write+0x116/0x250
[ 1358.511209] vfs_write+0x657/0x9b0
[ 1358.515826] ksys_write+0x116/0x250
[ 1358.520541] ? __ia32_sys_read+0xc0/0xc0
[ 1358.525741] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.531333] ? rcu_is_watching+0x15/0xd0
[ 1358.536537] do_syscall_64+0xf3/0x650
[ 1358.541447] ? trace_hardirqs_off+0xd/0x30
[ 1358.546846] ? exc_page_fault+0xda/0xf0
[ 1358.551956] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.558430] RIP: 0033:0x7f012e60e22e
[ 1358.563244] RSP: 002b:00007ffca7aad300 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.572549] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f012e60e22e
[ 1358.581365] RDX: 0000000000000001 RSI: 000055dc59c286b0 RDI: 0000000000000007
[ 1358.590175] RBP: 00007ffca7aad310 R08: 0000000000000000 R09: 0000000000000000
[ 1358.598992] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f012e486390
[ 1358.607808] R13: 000055dc59c286b0 R14: 0000000000000007 R15: 0000000000000003
[ 1358.616631]
[ 1358.619936] INFO: task python3:5851 is blocked on a mutex likely owned by task python3:3779.
[ 1358.630218] task:python3 state:D stack:25672 pid:3779 tgid:3779 ppid:3778 task_flags:0x400100 flags:0x00080802
[ 1358.643428] Call Trace:
[ 1358.646968]
[ 1358.650120] __schedule+0xcc3/0x19c0
[ 1358.654938] ? io_schedule_timeout+0x140/0x140
[ 1358.660719] ? find_held_lock+0x2b/0x80
[ 1358.665830] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.672012] schedule+0xd8/0x260
[ 1358.676432] schedule_timeout+0x13f/0x290
[ 1358.681734] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.688110] ? trace_contention_end+0xb3/0x1a0
[ 1358.693897] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.699594] ? ww_mutex_lock+0x160/0x160
[ 1358.704795] msleep+0x1b/0x30
[ 1358.708924] napi_set_threaded+0x3a5/0x440
[ 1358.714325] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.719822] netif_set_threaded+0x257/0x400
[ 1358.725319] threaded_store+0x1b8/0x220
[ 1358.730425] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.735729] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.741031] ? sysfs_kf_write+0x65/0x160
[ 1358.746237] ? sysfs_file_ops+0x140/0x140
[ 1358.751536] kernfs_fop_write_iter+0x335/0x510
[ 1358.757324] ? kernfs_ops+0xf0/0xf0
[ 1358.762038] new_sync_write+0x333/0x750
[ 1358.767149] ? __lock_acquire+0x508/0xc10
[ 1358.772438] ? new_sync_read+0x740/0x740
[ 1358.777646] ? lock_acquire.part.0+0xbc/0x260
[ 1358.783335] ? ksys_write+0x116/0x250
[ 1358.788253] vfs_write+0x657/0x9b0
[ 1358.792874] ksys_write+0x116/0x250
[ 1358.797593] ? __ia32_sys_read+0xc0/0xc0
[ 1358.802803] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.808393] ? rcu_is_watching+0x15/0xd0
[ 1358.813593] do_syscall_64+0xf3/0x650
[ 1358.818502] ? trace_hardirqs_off+0xd/0x30
[ 1358.823896] ? exc_page_fault+0xda/0xf0
[ 1358.829001] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.835469] RIP: 0033:0x7fb564c8d22e
[ 1358.840278] RSP: 002b:00007ffea3ecb0d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.849575] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb564c8d22e
[ 1358.858376] RDX: 0000000000000001 RSI: 0000561e58d266b0 RDI: 0000000000000007
[ 1358.867182] RBP: 00007ffea3ecb0e0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.875993] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fb564b05390
[ 1358.884806] R13: 0000561e58d266b0 R14: 0000000000000007 R15: 0000000000000003
[ 1358.893620]
[ 1358.896956] \x0aShowing all locks held in the system:
[ 1358.905498] 1 lock held by khungtaskd/184:
[ 1358.910894] #0: ffffffffb88fdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.922382] 4 locks held by python3/3779:
[ 1358.927678] #0: ff1100012fe993e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.937568] #1: ff11000188bfb880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.948531] #2: ff11000122d70b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.959688] #3: ff11000122e11d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.969784] 4 locks held by python3/5851:
[ 1358.975079] #0: ff1100012fe993e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.984971] #1: ff11000107138280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.995934] #2: ff11000122d70b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.007094] #3: ff11000122e11d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.017187] 2 locks held by 9/7687:
[ 1359.017192]
[ 1359.017193] =============================================\x0a