[ 790.998363] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 791.001367] nipa-hw-worker: [5/46] Retrying drivers/net:napi_threaded.py
[ 791.156928] kselftest: Running tests in drivers/net
[ 989.735222] INFO: task python3:5859 blocked for more than 122 seconds.
[ 989.743439] Not tainted 7.1.0-rc4-qitp-g9f07b9c9cb8b #1
[ 989.751261] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.760853] task:python3 state:D stack:25672 pid:5859 tgid:5859 ppid:5858 task_flags:0x400100 flags:0x00080000
[ 989.774061] Call Trace:
[ 989.777600]
[ 989.780753] __schedule+0xcc3/0x19c0
[ 989.785571] ? io_schedule_timeout+0x140/0x140
[ 989.791357] ? find_held_lock+0x2b/0x80
[ 989.796464] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.802350] schedule+0xd8/0x260
[ 989.806770] schedule_preempt_disabled+0x18/0x30
[ 989.812741] __mutex_lock+0x6af/0x2010
[ 989.817747] ? threaded_store+0x132/0x220
[ 989.823049] ? ww_mutex_lock+0x160/0x160
[ 989.828248] ? selinux_inode_free_security+0x330/0x330
[ 989.834817] ? cap_capable+0x1d6/0x3e0
[ 989.839827] ? threaded_store+0x132/0x220
[ 989.845125] threaded_store+0x132/0x220
[ 989.850230] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.855527] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.860828] ? sysfs_kf_write+0x65/0x160
[ 989.866031] ? sysfs_file_ops+0x140/0x140
[ 989.871326] kernfs_fop_write_iter+0x335/0x510
[ 989.877112] ? kernfs_ops+0xf0/0xf0
[ 989.881822] new_sync_write+0x333/0x750
[ 989.886925] ? __lock_acquire+0x508/0xc10
[ 989.892223] ? new_sync_read+0x740/0x740
[ 989.897424] ? lock_acquire.part.0+0xbc/0x260
[ 989.903110] ? ksys_write+0x116/0x250
[ 989.908030] vfs_write+0x657/0x9b0
[ 989.912645] ksys_write+0x116/0x250
[ 989.917356] ? __ia32_sys_read+0xc0/0xc0
[ 989.922558] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.928148] ? rcu_is_watching+0x15/0xd0
[ 989.933351] do_syscall_64+0xf3/0x650
[ 989.938257] ? trace_hardirqs_off+0xd/0x30
[ 989.943643] ? exc_page_fault+0xda/0xf0
[ 989.948745] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.955213] RIP: 0033:0x7fe9ae4de22e
[ 989.960027] RSP: 002b:00007ffc628de010 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.969324] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fe9ae4de22e
[ 989.978133] RDX: 0000000000000001 RSI: 000056042bf196b0 RDI: 0000000000000007
[ 989.986941] RBP: 00007ffc628de020 R08: 0000000000000000 R09: 0000000000000000
[ 989.995753] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fe9ae356390
[ 990.004559] R13: 000056042bf196b0 R14: 0000000000000007 R15: 0000000000000003
[ 990.013373]
[ 990.016664] INFO: task python3:5859 is blocked on a mutex likely owned by task python3:3806.
[ 990.026938] task:python3 state:D stack:25320 pid:3806 tgid:3806 ppid:3805 task_flags:0x400100 flags:0x00080802
[ 990.040139] Call Trace:
[ 990.043680]
[ 990.046830] __schedule+0xcc3/0x19c0
[ 990.051640] ? io_schedule_timeout+0x140/0x140
[ 990.057422] ? find_held_lock+0x2b/0x80
[ 990.062526] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.068706] schedule+0xd8/0x260
[ 990.073126] schedule_timeout+0x13f/0x290
[ 990.078425] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.084793] ? trace_contention_end+0xb3/0x1a0
[ 990.090578] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.096259] ? ww_mutex_lock+0x160/0x160
[ 990.101458] msleep+0x1b/0x30
[ 990.105573] napi_set_threaded+0x3a5/0x440
[ 990.110970] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.116467] netif_set_threaded+0x257/0x400
[ 990.121964] threaded_store+0x1b8/0x220
[ 990.127073] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.132369] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.137667] ? sysfs_kf_write+0x65/0x160
[ 990.142857] ? sysfs_file_ops+0x140/0x140
[ 990.148154] kernfs_fop_write_iter+0x335/0x510
[ 990.153932] ? kernfs_ops+0xf0/0xf0
[ 990.158645] new_sync_write+0x333/0x750
[ 990.163749] ? __lock_acquire+0x508/0xc10
[ 990.169046] ? new_sync_read+0x740/0x740
[ 990.174238] ? lock_acquire.part.0+0xbc/0x260
[ 990.179923] ? ksys_write+0x116/0x250
[ 990.184834] vfs_write+0x657/0x9b0
[ 990.189448] ksys_write+0x116/0x250
[ 990.194161] ? __ia32_sys_read+0xc0/0xc0
[ 990.199362] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.204942] ? rcu_is_watching+0x15/0xd0
[ 990.210144] do_syscall_64+0xf3/0x650
[ 990.215052] ? irq_exit_rcu+0x1a/0x30
[ 990.219959] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.226425] RIP: 0033:0x7f4326d7322e
[ 990.231232] RSP: 002b:00007fff5e8a8b80 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.240531] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f4326d7322e
[ 990.249343] RDX: 0000000000000001 RSI: 0000563f5fd406b0 RDI: 0000000000000007
[ 990.258152] RBP: 00007fff5e8a8b90 R08: 0000000000000000 R09: 0000000000000000
[ 990.266959] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f4326beb390
[ 990.275769] R13: 0000563f5fd406b0 R14: 0000000000000007 R15: 0000000000000003
[ 990.284580]
[ 990.287838] \x0aShowing all locks held in the system:
[ 990.296377] 1 lock held by khungtaskd/185:
[ 990.301772] #0: ffffffffb3cfdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.313247] 3 locks held by systemd-journal/801:
[ 990.319225] #0: ff1100180e5372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 990.329116] #1: ffffffffb3f029a0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 990.341932] #2: ff110001a75ea4f0 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 990.353489] 4 locks held by python3/3806:
[ 990.358783] #0: ff1100012e4643e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.368675] #1: ff1100017e0a2c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.379637] #2: ff1100012154eb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.390796] #3: ff11000121619d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.400899] 4 locks held by python3/5859:
[ 990.406192] #0: ff1100012e4643e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.416080] #1: ff110001b7242c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.427043] #2: ff1100012154eb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.438200] #3: ff11000121619d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.448278]
[ 990.450744] =============================================\x0a
[ 1063.461375] kworker/u130:7 (848) used greatest stack depth: 21936 bytes left
[ 1112.613680] INFO: task python3:5859 blocked for more than 245 seconds.
[ 1112.621830] Not tainted 7.1.0-rc4-qitp-g9f07b9c9cb8b #1
[ 1112.629081] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.638667] task:python3 state:D stack:25672 pid:5859 tgid:5859 ppid:5858 task_flags:0x400100 flags:0x00080000
[ 1112.651871] Call Trace:
[ 1112.655411]
[ 1112.658563] __schedule+0xcc3/0x19c0
[ 1112.663379] ? io_schedule_timeout+0x140/0x140
[ 1112.669164] ? find_held_lock+0x2b/0x80
[ 1112.674267] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.680150] schedule+0xd8/0x260
[ 1112.684556] schedule_preempt_disabled+0x18/0x30
[ 1112.690532] __mutex_lock+0x6af/0x2010
[ 1112.695535] ? threaded_store+0x132/0x220
[ 1112.700834] ? ww_mutex_lock+0x160/0x160
[ 1112.706033] ? selinux_inode_free_security+0x330/0x330
[ 1112.712603] ? cap_capable+0x1d6/0x3e0
[ 1112.717607] ? threaded_store+0x132/0x220
[ 1112.722904] threaded_store+0x132/0x220
[ 1112.728006] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.733303] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.738603] ? sysfs_kf_write+0x65/0x160
[ 1112.743805] ? sysfs_file_ops+0x140/0x140
[ 1112.749101] kernfs_fop_write_iter+0x335/0x510
[ 1112.754883] ? kernfs_ops+0xf0/0xf0
[ 1112.759593] new_sync_write+0x333/0x750
[ 1112.764688] ? __lock_acquire+0x508/0xc10
[ 1112.769984] ? new_sync_read+0x740/0x740
[ 1112.775182] ? lock_acquire.part.0+0xbc/0x260
[ 1112.780868] ? ksys_write+0x116/0x250
[ 1112.785778] vfs_write+0x657/0x9b0
[ 1112.790390] ksys_write+0x116/0x250
[ 1112.795102] ? __ia32_sys_read+0xc0/0xc0
[ 1112.800291] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.805882] ? rcu_is_watching+0x15/0xd0
[ 1112.811082] do_syscall_64+0xf3/0x650
[ 1112.815986] ? trace_hardirqs_off+0xd/0x30
[ 1112.821381] ? exc_page_fault+0xda/0xf0
[ 1112.826484] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.832950] RIP: 0033:0x7fe9ae4de22e
[ 1112.837749] RSP: 002b:00007ffc628de010 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.847047] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fe9ae4de22e
[ 1112.855854] RDX: 0000000000000001 RSI: 000056042bf196b0 RDI: 0000000000000007
[ 1112.864660] RBP: 00007ffc628de020 R08: 0000000000000000 R09: 0000000000000000
[ 1112.873467] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fe9ae356390
[ 1112.882264] R13: 000056042bf196b0 R14: 0000000000000007 R15: 0000000000000003
[ 1112.891076]
[ 1112.894367] INFO: task python3:5859 is blocked on a mutex likely owned by task python3:3806.
[ 1112.904639] task:python3 state:D stack:25320 pid:3806 tgid:3806 ppid:3805 task_flags:0x400100 flags:0x00080802
[ 1112.917839] Call Trace:
[ 1112.921379]
[ 1112.924527] __schedule+0xcc3/0x19c0
[ 1112.929336] ? io_schedule_timeout+0x140/0x140
[ 1112.935117] ? find_held_lock+0x2b/0x80
[ 1112.940218] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.946397] schedule+0xd8/0x260
[ 1112.950816] schedule_timeout+0x13f/0x290
[ 1112.956110] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.962476] ? trace_contention_end+0xb3/0x1a0
[ 1112.968260] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.973948] ? ww_mutex_lock+0x160/0x160
[ 1112.979145] msleep+0x1b/0x30
[ 1112.983267] napi_set_threaded+0x3a5/0x440
[ 1112.988664] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.994160] netif_set_threaded+0x257/0x400
[ 1112.999651] threaded_store+0x1b8/0x220
[ 1113.004754] ? xps_rxqs_store+0x2e0/0x2e0
[ 1113.010048] ? sysfs_file_kobj+0xbd/0x1e0
[ 1113.015344] ? sysfs_kf_write+0x65/0x160
[ 1113.020541] ? sysfs_file_ops+0x140/0x140
[ 1113.025835] kernfs_fop_write_iter+0x335/0x510
[ 1113.031620] ? kernfs_ops+0xf0/0xf0
[ 1113.036328] new_sync_write+0x333/0x750
[ 1113.041428] ? __lock_acquire+0x508/0xc10
[ 1113.046724] ? new_sync_read+0x740/0x740
[ 1113.051923] ? lock_acquire.part.0+0xbc/0x260
[ 1113.057607] ? ksys_write+0x116/0x250
[ 1113.062524] vfs_write+0x657/0x9b0
[ 1113.067137] ksys_write+0x116/0x250
[ 1113.071848] ? __ia32_sys_read+0xc0/0xc0
[ 1113.077047] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.082637] ? rcu_is_watching+0x15/0xd0
[ 1113.087837] do_syscall_64+0xf3/0x650
[ 1113.092744] ? irq_exit_rcu+0x1a/0x30
[ 1113.097642] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.104108] RIP: 0033:0x7f4326d7322e
[ 1113.108913] RSP: 002b:00007fff5e8a8b80 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.118208] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f4326d7322e
[ 1113.127014] RDX: 0000000000000001 RSI: 0000563f5fd406b0 RDI: 0000000000000007
[ 1113.135823] RBP: 00007fff5e8a8b90 R08: 0000000000000000 R09: 0000000000000000
[ 1113.144631] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f4326beb390
[ 1113.153436] R13: 0000563f5fd406b0 R14: 0000000000000007 R15: 0000000000000003
[ 1113.162247]
[ 1113.165508] \x0aShowing all locks held in the system:
[ 1113.174046] 1 lock held by khungtaskd/185:
[ 1113.179440] #0: ffffffffb3cfdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.190920] 4 locks held by python3/3806:
[ 1113.196214] #0: ff1100012e4643e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.206102] #1: ff1100017e0a2c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.217063] #2: ff1100012154eb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.228212] #3: ff11000121619d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.238321] 4 locks held by python3/5859:
[ 1113.243618] #0: ff1100012e4643e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.253506] #1: ff110001b7242c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.264456] #2: ff1100012154eb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.275612] #3: ff11000121619d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.285701] 7 locks held by sshd-auth/6899:
[ 1113.291191] #0: ff1100180dd372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1113.301079] #1: ffffffffb3cfdde0 (rcu_read_lock){....}-{1:3}, at: __update_idle_core+0x5a/0x700
[ 1113.311750] #2: ffffffffb73051f8 (&obj_hash[i].lock){-.-.}-{2:2}, at: debug_object_activate+0x141/0x4b0
[ 1113.323198] #3: ff1100014ca8ad68 (&p->pi_lock){-.-.}-{2:2}, at: try_to_wake_up+0xad/0xfb0
[ 1113.333281] #4: ff1100180dcb72e0 (&rq->__lock){-.-.}-{2:2}, at: ___task_rq_lock+0xf7/0x4b0
[ 1113.343462] #5: ff110001ed338790 (key){....}-{2:2}, at: percpu_counter_add_batch+0x89/0x280
[ 1113.353807] #6: ff1100012116c318 (_xmit_ETHER){+.-.}-{3:3}, at: sch_direct_xmit+0x2e5/0x1100
[ 1113.364189]
[ 1113.366653] =============================================\x0a
[ 1235.495861] INFO: task python3:5859 blocked for more than 368 seconds.
[ 1235.504010] Not tainted 7.1.0-rc4-qitp-g9f07b9c9cb8b #1
[ 1235.511262] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.520851] task:python3 state:D stack:25672 pid:5859 tgid:5859 ppid:5858 task_flags:0x400100 flags:0x00080002
[ 1235.534047] Call Trace:
[ 1235.537587]
[ 1235.540741] __schedule+0xcc3/0x19c0
[ 1235.545560] ? io_schedule_timeout+0x140/0x140
[ 1235.551348] ? find_held_lock+0x2b/0x80
[ 1235.556455] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.562341] schedule+0xd8/0x260
[ 1235.566758] schedule_preempt_disabled+0x18/0x30
[ 1235.572740] __mutex_lock+0x6af/0x2010
[ 1235.577744] ? threaded_store+0x132/0x220
[ 1235.583044] ? ww_mutex_lock+0x160/0x160
[ 1235.588244] ? selinux_inode_free_security+0x330/0x330
[ 1235.594815] ? cap_capable+0x1d6/0x3e0
[ 1235.599823] ? threaded_store+0x132/0x220
[ 1235.605122] threaded_store+0x132/0x220
[ 1235.610223] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.615523] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.620822] ? sysfs_kf_write+0x65/0x160
[ 1235.626023] ? sysfs_file_ops+0x140/0x140
[ 1235.631321] kernfs_fop_write_iter+0x335/0x510
[ 1235.637104] ? kernfs_ops+0xf0/0xf0
[ 1235.641816] new_sync_write+0x333/0x750
[ 1235.646921] ? __lock_acquire+0x508/0xc10
[ 1235.652217] ? new_sync_read+0x740/0x740
[ 1235.657420] ? lock_acquire.part.0+0xbc/0x260
[ 1235.663107] ? ksys_write+0x116/0x250
[ 1235.668016] vfs_write+0x657/0x9b0
[ 1235.672634] ksys_write+0x116/0x250
[ 1235.677346] ? __ia32_sys_read+0xc0/0xc0
[ 1235.682535] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.688126] ? rcu_is_watching+0x15/0xd0
[ 1235.693328] do_syscall_64+0xf3/0x650
[ 1235.698237] ? trace_hardirqs_off+0xd/0x30
[ 1235.703633] ? exc_page_fault+0xda/0xf0
[ 1235.708736] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.715203] RIP: 0033:0x7fe9ae4de22e
[ 1235.720013] RSP: 002b:00007ffc628de010 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.729312] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fe9ae4de22e
[ 1235.738122] RDX: 0000000000000001 RSI: 000056042bf196b0 RDI: 0000000000000007
[ 1235.746932] RBP: 00007ffc628de020 R08: 0000000000000000 R09: 0000000000000000
[ 1235.755742] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fe9ae356390
[ 1235.764549] R13: 000056042bf196b0 R14: 0000000000000007 R15: 0000000000000003
[ 1235.773361]
[ 1235.776649] INFO: task python3:5859 is blocked on a mutex likely owned by task python3:3806.
[ 1235.786923] task:python3 state:D stack:25320 pid:3806 tgid:3806 ppid:3805 task_flags:0x400100 flags:0x00080802
[ 1235.800127] Call Trace:
[ 1235.803665]
[ 1235.806817] __schedule+0xcc3/0x19c0
[ 1235.811629] ? io_schedule_timeout+0x140/0x140
[ 1235.817414] ? find_held_lock+0x2b/0x80
[ 1235.822510] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.828686] schedule+0xd8/0x260
[ 1235.833106] schedule_timeout+0x13f/0x290
[ 1235.838394] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.844764] ? trace_contention_end+0xb3/0x1a0
[ 1235.850542] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.856234] ? ww_mutex_lock+0x160/0x160
[ 1235.861435] msleep+0x1b/0x30
[ 1235.865562] napi_set_threaded+0x3a5/0x440
[ 1235.870952] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.876446] netif_set_threaded+0x257/0x400
[ 1235.881941] threaded_store+0x1b8/0x220
[ 1235.887044] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.892333] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.897619] ? sysfs_kf_write+0x65/0x160
[ 1235.902819] ? sysfs_file_ops+0x140/0x140
[ 1235.908106] kernfs_fop_write_iter+0x335/0x510
[ 1235.913891] ? kernfs_ops+0xf0/0xf0
[ 1235.918602] new_sync_write+0x333/0x750
[ 1235.923710] ? __lock_acquire+0x508/0xc10
[ 1235.929007] ? new_sync_read+0x740/0x740
[ 1235.934200] ? lock_acquire.part.0+0xbc/0x260
[ 1235.939887] ? ksys_write+0x116/0x250
[ 1235.944787] vfs_write+0x657/0x9b0
[ 1235.949401] ksys_write+0x116/0x250
[ 1235.954116] ? __ia32_sys_read+0xc0/0xc0
[ 1235.959314] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.964902] ? rcu_is_watching+0x15/0xd0
[ 1235.970102] do_syscall_64+0xf3/0x650
[ 1235.975009] ? irq_exit_rcu+0x1a/0x30
[ 1235.979918] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.986383] RIP: 0033:0x7f4326d7322e
[ 1235.991192] RSP: 002b:00007fff5e8a8b80 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1236.000491] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f4326d7322e
[ 1236.009301] RDX: 0000000000000001 RSI: 0000563f5fd406b0 RDI: 0000000000000007
[ 1236.018111] RBP: 00007fff5e8a8b90 R08: 0000000000000000 R09: 0000000000000000
[ 1236.026918] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f4326beb390
[ 1236.035726] R13: 0000563f5fd406b0 R14: 0000000000000007 R15: 0000000000000003
[ 1236.044539]
[ 1236.047805] \x0aShowing all locks held in the system:
[ 1236.056346] 1 lock held by khungtaskd/185:
[ 1236.061742] #0: ffffffffb3cfdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.073214] 2 locks held by systemd-journal/801:
[ 1236.079195] #0: ff1100180e2b72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1236.089083] #1: ffffffffb3f029a0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1236.101909] 4 locks held by python3/3806:
[ 1236.107203] #0: ff1100012e4643e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.117093] #1: ff1100017e0a2c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.128053] #2: ff1100012154eb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.139205] #3: ff11000121619d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.149299] 4 locks held by python3/5859:
[ 1236.154595] #0: ff1100012e4643e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.164484] #1: ff110001b7242c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.175444] #2: ff1100012154eb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.186599] #3: ff11000121619d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.196689]
[ 1236.199156] =============================================\x0a
[ 1358.376515] INFO: task python3:5859 blocked for more than 491 seconds.
[ 1358.384662] Not tainted 7.1.0-rc4-qitp-g9f07b9c9cb8b #1
[ 1358.391913] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.401501] task:python3 state:D stack:25672 pid:5859 tgid:5859 ppid:5858 task_flags:0x400100 flags:0x00080002
[ 1358.414707] Call Trace:
[ 1358.418244]
[ 1358.421396] __schedule+0xcc3/0x19c0
[ 1358.426213] ? io_schedule_timeout+0x140/0x140
[ 1358.431997] ? find_held_lock+0x2b/0x80
[ 1358.437103] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.443003] schedule+0xd8/0x260
[ 1358.447427] schedule_preempt_disabled+0x18/0x30
[ 1358.453406] __mutex_lock+0x6af/0x2010
[ 1358.458478] ? threaded_store+0x132/0x220
[ 1358.463778] ? ww_mutex_lock+0x160/0x160
[ 1358.468976] ? selinux_inode_free_security+0x330/0x330
[ 1358.475545] ? cap_capable+0x1d6/0x3e0
[ 1358.480551] ? threaded_store+0x132/0x220
[ 1358.485923] threaded_store+0x132/0x220
[ 1358.491024] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.496320] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.501607] ? sysfs_kf_write+0x65/0x160
[ 1358.506805] ? sysfs_file_ops+0x140/0x140
[ 1358.512100] kernfs_fop_write_iter+0x335/0x510
[ 1358.517882] ? kernfs_ops+0xf0/0xf0
[ 1358.522593] new_sync_write+0x333/0x750
[ 1358.527694] ? __lock_acquire+0x508/0xc10
[ 1358.532991] ? new_sync_read+0x740/0x740
[ 1358.538193] ? lock_acquire.part.0+0xbc/0x260
[ 1358.543878] ? ksys_write+0x116/0x250
[ 1358.548786] vfs_write+0x657/0x9b0
[ 1358.553399] ksys_write+0x116/0x250
[ 1358.558111] ? __ia32_sys_read+0xc0/0xc0
[ 1358.563302] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.568890] ? rcu_is_watching+0x15/0xd0
[ 1358.574092] do_syscall_64+0xf3/0x650
[ 1358.578998] ? trace_hardirqs_off+0xd/0x30
[ 1358.584383] ? exc_page_fault+0xda/0xf0
[ 1358.589487] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.595954] RIP: 0033:0x7fe9ae4de22e
[ 1358.600762] RSP: 002b:00007ffc628de010 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.610059] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fe9ae4de22e
[ 1358.618857] RDX: 0000000000000001 RSI: 000056042bf196b0 RDI: 0000000000000007
[ 1358.627662] RBP: 00007ffc628de020 R08: 0000000000000000 R09: 0000000000000000
[ 1358.636472] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fe9ae356390
[ 1358.645278] R13: 000056042bf196b0 R14: 0000000000000007 R15: 0000000000000003
[ 1358.654089]
[ 1358.657464] INFO: task python3:5859 is blocked on a mutex likely owned by task python3:3806.
[ 1358.667736] task:python3 state:D stack:25320 pid:3806 tgid:3806 ppid:3805 task_flags:0x400100 flags:0x00080802
[ 1358.680936] Call Trace:
[ 1358.684476]
[ 1358.687623] __schedule+0xcc3/0x19c0
[ 1358.692499] ? io_schedule_timeout+0x140/0x140
[ 1358.698284] ? find_held_lock+0x2b/0x80
[ 1358.703453] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.709629] schedule+0xd8/0x260
[ 1358.714045] schedule_timeout+0x13f/0x290
[ 1358.719341] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.725708] ? trace_contention_end+0xb3/0x1a0
[ 1358.731491] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.737180] ? ww_mutex_lock+0x160/0x160
[ 1358.742378] msleep+0x1b/0x30
[ 1358.746500] napi_set_threaded+0x3a5/0x440
[ 1358.751896] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.757390] netif_set_threaded+0x257/0x400
[ 1358.762881] threaded_store+0x1b8/0x220
[ 1358.767984] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.773282] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.778569] ? sysfs_kf_write+0x65/0x160
[ 1358.783767] ? sysfs_file_ops+0x140/0x140
[ 1358.789062] kernfs_fop_write_iter+0x335/0x510
[ 1358.794846] ? kernfs_ops+0xf0/0xf0
[ 1358.799557] new_sync_write+0x333/0x750
[ 1358.804659] ? __lock_acquire+0x508/0xc10
[ 1358.809956] ? new_sync_read+0x740/0x740
[ 1358.815157] ? lock_acquire.part.0+0xbc/0x260
[ 1358.820843] ? ksys_write+0x116/0x250
[ 1358.825751] vfs_write+0x657/0x9b0
[ 1358.830367] ksys_write+0x116/0x250
[ 1358.835076] ? __ia32_sys_read+0xc0/0xc0
[ 1358.840274] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.845927] ? rcu_is_watching+0x15/0xd0
[ 1358.851125] do_syscall_64+0xf3/0x650
[ 1358.856032] ? irq_exit_rcu+0x1a/0x30
[ 1358.860928] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.867461] RIP: 0033:0x7f4326d7322e
[ 1358.872268] RSP: 002b:00007fff5e8a8b80 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.881564] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f4326d7322e
[ 1358.890370] RDX: 0000000000000001 RSI: 0000563f5fd406b0 RDI: 0000000000000007
[ 1358.899177] RBP: 00007fff5e8a8b90 R08: 0000000000000000 R09: 0000000000000000
[ 1358.907986] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f4326beb390
[ 1358.916791] R13: 0000563f5fd406b0 R14: 0000000000000007 R15: 0000000000000003
[ 1358.925602]
[ 1358.928868] \x0aShowing all locks held in the system:
[ 1358.937401] 1 lock held by khungtaskd/185:
[ 1358.942785] #0: ffffffffb3cfdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.954258] 3 locks held by systemd-journal/801:
[ 1358.960236] #0: ff1100180df372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1358.970123] #1: ffffffffb3f029a0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1358.982937] #2: ff110001a75ea650 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 1358.994485] 4 locks held by python3/3806:
[ 1358.999781] #0: ff1100012e4643e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1359.009667] #1: ff1100017e0a2c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1359.020628] #2: ff1100012154eb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.031783] #3: ff11000121619d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.041882] 4 locks held by python3/5859:
[ 1359.047168] #0: ff1100012e4643e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1359.057055] #1: ff110001b7242c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1359.068016] #2: ff1100012154eb50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.079173] #3: ff11000121619d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.089262] 1 lock held by systemd/7708:
[ 1359.094461]
[ 1359.096929] =============================================\x0a