[ 792.757198] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 792.759610] nipa-hw-worker: [5/47] Retrying drivers/net:napi_threaded.py
[ 792.890409] kselftest: Running tests in drivers/net
[ 989.646581] INFO: task python3:5879 blocked for more than 122 seconds.
[ 989.654800] Not tainted 7.1.0-rc3-llwx-gef069d5c7dcc #1
[ 989.662576] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.672165] task:python3 state:D stack:25672 pid:5879 tgid:5879 ppid:5878 task_flags:0x400100 flags:0x00080000
[ 989.685370] Call Trace:
[ 989.688911]
[ 989.692062] __schedule+0xcc3/0x19c0
[ 989.696882] ? io_schedule_timeout+0x140/0x140
[ 989.702669] ? find_held_lock+0x2b/0x80
[ 989.707777] schedule+0xd4/0x260
[ 989.712197] schedule_preempt_disabled+0x18/0x30
[ 989.718175] __mutex_lock+0x6a3/0x2000
[ 989.723179] ? threaded_store+0x132/0x220
[ 989.728479] ? ww_mutex_lock+0x160/0x160
[ 989.733677] ? selinux_inode_free_security+0x330/0x330
[ 989.740249] ? cap_capable+0x1d6/0x3e0
[ 989.745259] ? threaded_store+0x132/0x220
[ 989.750560] threaded_store+0x132/0x220
[ 989.755667] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.760962] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.766261] ? sysfs_kf_write+0x65/0x160
[ 989.771456] ? sysfs_file_ops+0x140/0x140
[ 989.776751] kernfs_fop_write_iter+0x335/0x510
[ 989.782537] ? kernfs_ops+0xf0/0xf0
[ 989.787251] new_sync_write+0x333/0x750
[ 989.792356] ? __lock_acquire+0x508/0xc10
[ 989.797654] ? new_sync_read+0x740/0x740
[ 989.802846] ? lock_acquire.part.0+0xbc/0x260
[ 989.808534] ? ksys_write+0x116/0x250
[ 989.813446] vfs_write+0x657/0x9b0
[ 989.818062] ksys_write+0x116/0x250
[ 989.822773] ? __ia32_sys_read+0xc0/0xc0
[ 989.827973] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.833553] ? rcu_is_watching+0x15/0xd0
[ 989.838758] do_syscall_64+0xf3/0x650
[ 989.843665] ? trace_hardirqs_off+0xd/0x30
[ 989.849059] ? exc_page_fault+0xda/0xf0
[ 989.854161] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.860629] RIP: 0033:0x7fd87b04222e
[ 989.865439] RSP: 002b:00007ffd10435b80 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.874736] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fd87b04222e
[ 989.883544] RDX: 0000000000000001 RSI: 000056307ec4dcd0 RDI: 0000000000000007
[ 989.892350] RBP: 00007ffd10435b90 R08: 0000000000000000 R09: 0000000000000000
[ 989.901157] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fd87aeba390
[ 989.909965] R13: 000056307ec4dcd0 R14: 0000000000000007 R15: 0000000000000003
[ 989.918778]
[ 989.922083] INFO: task python3:5879 is blocked on a mutex likely owned by task python3:3833.
[ 989.932355] task:python3 state:D stack:25584 pid:3833 tgid:3833 ppid:3832 task_flags:0x400100 flags:0x00080802
[ 989.945560] Call Trace:
[ 989.949100]
[ 989.952249] __schedule+0xcc3/0x19c0
[ 989.957058] ? io_schedule_timeout+0x140/0x140
[ 989.962845] ? find_held_lock+0x2b/0x80
[ 989.967947] ? __try_to_del_timer_sync+0x155/0x1f0
[ 989.974128] schedule+0xd4/0x260
[ 989.978547] schedule_timeout+0x13f/0x290
[ 989.983837] ? hrtimer_nanosleep_restart+0x270/0x270
[ 989.990209] ? trace_contention_end+0xb3/0x1a0
[ 989.995993] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.001684] ? ww_mutex_lock+0x160/0x160
[ 990.006885] msleep+0x1b/0x30
[ 990.011013] napi_set_threaded+0x3a5/0x440
[ 990.016415] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.021910] netif_set_threaded+0x257/0x400
[ 990.027406] threaded_store+0x1b8/0x220
[ 990.032512] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.037808] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.043105] ? sysfs_kf_write+0x65/0x160
[ 990.048306] ? sysfs_file_ops+0x140/0x140
[ 990.053604] kernfs_fop_write_iter+0x335/0x510
[ 990.059388] ? kernfs_ops+0xf0/0xf0
[ 990.064100] new_sync_write+0x333/0x750
[ 990.069205] ? __lock_acquire+0x508/0xc10
[ 990.074500] ? new_sync_read+0x740/0x740
[ 990.079700] ? lock_acquire.part.0+0xbc/0x260
[ 990.085387] ? ksys_write+0x116/0x250
[ 990.090297] vfs_write+0x657/0x9b0
[ 990.094916] ksys_write+0x116/0x250
[ 990.099626] ? __ia32_sys_read+0xc0/0xc0
[ 990.104825] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.110418] ? rcu_is_watching+0x15/0xd0
[ 990.115620] do_syscall_64+0xf3/0x650
[ 990.120525] ? trace_hardirqs_off+0xd/0x30
[ 990.125921] ? exc_page_fault+0xda/0xf0
[ 990.131024] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.137481] RIP: 0033:0x7feeb537b22e
[ 990.142293] RSP: 002b:00007ffe3dba00c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.151590] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007feeb537b22e
[ 990.160396] RDX: 0000000000000001 RSI: 00005609330b1cd0 RDI: 0000000000000007
[ 990.169204] RBP: 00007ffe3dba00d0 R08: 0000000000000000 R09: 0000000000000000
[ 990.178014] R10: 0000000000000000 R11: 0000000000000202 R12: 00007feeb51f3390
[ 990.186819] R13: 00005609330b1cd0 R14: 0000000000000007 R15: 0000000000000003
[ 990.195634]
[ 990.198893] \x0aShowing all locks held in the system:
[ 990.207439] 1 lock held by khungtaskd/185:
[ 990.212833] #0: ffffffffb1efe060 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.224309] 2 locks held by systemd-journal/805:
[ 990.230289] #0: ff1100180e2372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 990.240177] #1: ffffffffb21033a0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 990.253006] 4 locks held by python3/3833:
[ 990.258300] #0: ff1100012eb5c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.268190] #1: ff11000179ab3e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.279154] #2: ff11000121344b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.290310] #3: ff110001213e9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.300406] 4 locks held by python3/5879:
[ 990.305705] #0: ff1100012eb5c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.315596] #1: ff110001cc826880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.326556] #2: ff11000121344b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.337712] #3: ff110001213e9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.347802]
[ 990.350260] =============================================\x0a
[ 1112.527662] INFO: task python3:5879 blocked for more than 245 seconds.
[ 1112.535809] Not tainted 7.1.0-rc3-llwx-gef069d5c7dcc #1
[ 1112.543059] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.552647] task:python3 state:D stack:25672 pid:5879 tgid:5879 ppid:5878 task_flags:0x400100 flags:0x00080000
[ 1112.565852] Call Trace:
[ 1112.569394]
[ 1112.572544] __schedule+0xcc3/0x19c0
[ 1112.577363] ? io_schedule_timeout+0x140/0x140
[ 1112.583150] ? find_held_lock+0x2b/0x80
[ 1112.588257] schedule+0xd4/0x260
[ 1112.592678] schedule_preempt_disabled+0x18/0x30
[ 1112.598658] __mutex_lock+0x6a3/0x2000
[ 1112.603654] ? threaded_store+0x132/0x220
[ 1112.608955] ? ww_mutex_lock+0x160/0x160
[ 1112.614153] ? selinux_inode_free_security+0x330/0x330
[ 1112.620726] ? cap_capable+0x1d6/0x3e0
[ 1112.625733] ? threaded_store+0x132/0x220
[ 1112.631030] threaded_store+0x132/0x220
[ 1112.636134] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.641430] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.646723] ? sysfs_kf_write+0x65/0x160
[ 1112.651924] ? sysfs_file_ops+0x140/0x140
[ 1112.657220] kernfs_fop_write_iter+0x335/0x510
[ 1112.663006] ? kernfs_ops+0xf0/0xf0
[ 1112.667716] new_sync_write+0x333/0x750
[ 1112.672819] ? __lock_acquire+0x508/0xc10
[ 1112.678118] ? new_sync_read+0x740/0x740
[ 1112.683319] ? lock_acquire.part.0+0xbc/0x260
[ 1112.689007] ? ksys_write+0x116/0x250
[ 1112.693918] vfs_write+0x657/0x9b0
[ 1112.698534] ksys_write+0x116/0x250
[ 1112.703246] ? __ia32_sys_read+0xc0/0xc0
[ 1112.708446] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.714035] ? rcu_is_watching+0x15/0xd0
[ 1112.719238] do_syscall_64+0xf3/0x650
[ 1112.724145] ? trace_hardirqs_off+0xd/0x30
[ 1112.729539] ? exc_page_fault+0xda/0xf0
[ 1112.734640] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.741100] RIP: 0033:0x7fd87b04222e
[ 1112.745909] RSP: 002b:00007ffd10435b80 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.755205] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fd87b04222e
[ 1112.764013] RDX: 0000000000000001 RSI: 000056307ec4dcd0 RDI: 0000000000000007
[ 1112.772820] RBP: 00007ffd10435b90 R08: 0000000000000000 R09: 0000000000000000
[ 1112.781628] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fd87aeba390
[ 1112.790437] R13: 000056307ec4dcd0 R14: 0000000000000007 R15: 0000000000000003
[ 1112.799251]
[ 1112.802541] INFO: task python3:5879 is blocked on a mutex likely owned by task python3:3833.
[ 1112.812815] task:python3 state:D stack:25584 pid:3833 tgid:3833 ppid:3832 task_flags:0x400100 flags:0x00080802
[ 1112.826018] Call Trace:
[ 1112.829559]
[ 1112.832708] __schedule+0xcc3/0x19c0
[ 1112.837520] ? io_schedule_timeout+0x140/0x140
[ 1112.843302] ? find_held_lock+0x2b/0x80
[ 1112.848404] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.854585] schedule+0xd4/0x260
[ 1112.859005] schedule_timeout+0x13f/0x290
[ 1112.864304] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.870672] ? trace_contention_end+0xb3/0x1a0
[ 1112.876456] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.882146] ? ww_mutex_lock+0x160/0x160
[ 1112.887344] msleep+0x1b/0x30
[ 1112.891468] napi_set_threaded+0x3a5/0x440
[ 1112.896866] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.902362] netif_set_threaded+0x257/0x400
[ 1112.907855] threaded_store+0x1b8/0x220
[ 1112.912949] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.918246] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.923544] ? sysfs_kf_write+0x65/0x160
[ 1112.928733] ? sysfs_file_ops+0x140/0x140
[ 1112.934028] kernfs_fop_write_iter+0x335/0x510
[ 1112.939813] ? kernfs_ops+0xf0/0xf0
[ 1112.944515] new_sync_write+0x333/0x750
[ 1112.949615] ? __lock_acquire+0x508/0xc10
[ 1112.954914] ? new_sync_read+0x740/0x740
[ 1112.960114] ? lock_acquire.part.0+0xbc/0x260
[ 1112.965791] ? ksys_write+0x116/0x250
[ 1112.970703] vfs_write+0x657/0x9b0
[ 1112.975330] ksys_write+0x116/0x250
[ 1112.980041] ? __ia32_sys_read+0xc0/0xc0
[ 1112.985243] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.990832] ? rcu_is_watching+0x15/0xd0
[ 1112.996035] do_syscall_64+0xf3/0x650
[ 1113.000940] ? trace_hardirqs_off+0xd/0x30
[ 1113.006333] ? exc_page_fault+0xda/0xf0
[ 1113.011435] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.017902] RIP: 0033:0x7feeb537b22e
[ 1113.022702] RSP: 002b:00007ffe3dba00c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.031992] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007feeb537b22e
[ 1113.040807] RDX: 0000000000000001 RSI: 00005609330b1cd0 RDI: 0000000000000007
[ 1113.049615] RBP: 00007ffe3dba00d0 R08: 0000000000000000 R09: 0000000000000000
[ 1113.058423] R10: 0000000000000000 R11: 0000000000000202 R12: 00007feeb51f3390
[ 1113.067222] R13: 00005609330b1cd0 R14: 0000000000000007 R15: 0000000000000003
[ 1113.076036]
[ 1113.079300] \x0aShowing all locks held in the system:
[ 1113.087839] 1 lock held by khungtaskd/185:
[ 1113.093235] #0: ffffffffb1efe060 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.104700] 2 locks held by systemd-journal/805:
[ 1113.110673] #0: ff1100180dab72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1113.120561] #1: ffffffffb21033a0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1113.133387] 4 locks held by python3/3833:
[ 1113.138685] #0: ff1100012eb5c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.148572] #1: ff11000179ab3e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.159535] #2: ff11000121344b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.170693] #3: ff110001213e9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.180787] 4 locks held by python3/5879:
[ 1113.186086] #0: ff1100012eb5c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.195978] #1: ff110001cc826880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.206941] #2: ff11000121344b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.218099] #3: ff110001213e9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.228187]
[ 1113.230652] =============================================\x0a
[ 1235.407750] INFO: task python3:5879 blocked for more than 368 seconds.
[ 1235.415897] Not tainted 7.1.0-rc3-llwx-gef069d5c7dcc #1
[ 1235.423147] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.432736] task:python3 state:D stack:25672 pid:5879 tgid:5879 ppid:5878 task_flags:0x400100 flags:0x00080002
[ 1235.445933] Call Trace:
[ 1235.449473]
[ 1235.452621] __schedule+0xcc3/0x19c0
[ 1235.457434] ? io_schedule_timeout+0x140/0x140
[ 1235.463219] ? find_held_lock+0x2b/0x80
[ 1235.468325] schedule+0xd4/0x260
[ 1235.472745] schedule_preempt_disabled+0x18/0x30
[ 1235.478713] __mutex_lock+0x6a3/0x2000
[ 1235.483718] ? threaded_store+0x132/0x220
[ 1235.489015] ? ww_mutex_lock+0x160/0x160
[ 1235.494211] ? selinux_inode_free_security+0x330/0x330
[ 1235.500782] ? cap_capable+0x1d6/0x3e0
[ 1235.505787] ? threaded_store+0x132/0x220
[ 1235.511073] threaded_store+0x132/0x220
[ 1235.516176] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.521463] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.526753] ? sysfs_kf_write+0x65/0x160
[ 1235.531955] ? sysfs_file_ops+0x140/0x140
[ 1235.537252] kernfs_fop_write_iter+0x335/0x510
[ 1235.543037] ? kernfs_ops+0xf0/0xf0
[ 1235.547746] new_sync_write+0x333/0x750
[ 1235.552848] ? __lock_acquire+0x508/0xc10
[ 1235.558141] ? new_sync_read+0x740/0x740
[ 1235.563341] ? lock_acquire.part.0+0xbc/0x260
[ 1235.569024] ? ksys_write+0x116/0x250
[ 1235.573933] vfs_write+0x657/0x9b0
[ 1235.578549] ksys_write+0x116/0x250
[ 1235.583262] ? __ia32_sys_read+0xc0/0xc0
[ 1235.588461] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.594050] ? rcu_is_watching+0x15/0xd0
[ 1235.599243] do_syscall_64+0xf3/0x650
[ 1235.604150] ? trace_hardirqs_off+0xd/0x30
[ 1235.609544] ? exc_page_fault+0xda/0xf0
[ 1235.614645] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.621113] RIP: 0033:0x7fd87b04222e
[ 1235.625924] RSP: 002b:00007ffd10435b80 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.635220] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fd87b04222e
[ 1235.644028] RDX: 0000000000000001 RSI: 000056307ec4dcd0 RDI: 0000000000000007
[ 1235.652836] RBP: 00007ffd10435b90 R08: 0000000000000000 R09: 0000000000000000
[ 1235.661633] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fd87aeba390
[ 1235.670442] R13: 000056307ec4dcd0 R14: 0000000000000007 R15: 0000000000000003
[ 1235.679255]
[ 1235.682532] INFO: task python3:5879 is blocked on a mutex likely owned by task python3:3833.
[ 1235.692803] task:python3 state:D stack:25584 pid:3833 tgid:3833 ppid:3832 task_flags:0x400100 flags:0x00080802
[ 1235.706005] Call Trace:
[ 1235.709542]
[ 1235.712692] __schedule+0xcc3/0x19c0
[ 1235.717504] ? io_schedule_timeout+0x140/0x140
[ 1235.723286] ? find_held_lock+0x2b/0x80
[ 1235.728386] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.734563] schedule+0xd4/0x260
[ 1235.738980] schedule_timeout+0x13f/0x290
[ 1235.744277] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.750645] ? trace_contention_end+0xb3/0x1a0
[ 1235.756431] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.762122] ? ww_mutex_lock+0x160/0x160
[ 1235.767322] msleep+0x1b/0x30
[ 1235.771438] napi_set_threaded+0x3a5/0x440
[ 1235.776835] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.782329] netif_set_threaded+0x257/0x400
[ 1235.787822] threaded_store+0x1b8/0x220
[ 1235.792921] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.798216] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.803509] ? sysfs_kf_write+0x65/0x160
[ 1235.808708] ? sysfs_file_ops+0x140/0x140
[ 1235.814005] kernfs_fop_write_iter+0x335/0x510
[ 1235.819790] ? kernfs_ops+0xf0/0xf0
[ 1235.824490] new_sync_write+0x333/0x750
[ 1235.829594] ? __lock_acquire+0x508/0xc10
[ 1235.834888] ? new_sync_read+0x740/0x740
[ 1235.840089] ? lock_acquire.part.0+0xbc/0x260
[ 1235.845775] ? ksys_write+0x116/0x250
[ 1235.850683] vfs_write+0x657/0x9b0
[ 1235.855296] ksys_write+0x116/0x250
[ 1235.860009] ? __ia32_sys_read+0xc0/0xc0
[ 1235.865206] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.870792] ? rcu_is_watching+0x15/0xd0
[ 1235.875991] do_syscall_64+0xf3/0x650
[ 1235.880894] ? trace_hardirqs_off+0xd/0x30
[ 1235.886289] ? exc_page_fault+0xda/0xf0
[ 1235.891391] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.897855] RIP: 0033:0x7feeb537b22e
[ 1235.902664] RSP: 002b:00007ffe3dba00c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.911951] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007feeb537b22e
[ 1235.920756] RDX: 0000000000000001 RSI: 00005609330b1cd0 RDI: 0000000000000007
[ 1235.929561] RBP: 00007ffe3dba00d0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.938366] R10: 0000000000000000 R11: 0000000000000202 R12: 00007feeb51f3390
[ 1235.947170] R13: 00005609330b1cd0 R14: 0000000000000007 R15: 0000000000000003
[ 1235.955981]
[ 1235.959246] \x0aShowing all locks held in the system:
[ 1235.967775] 1 lock held by khungtaskd/185:
[ 1235.973167] #0: ffffffffb1efe060 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1235.984641] 2 locks held by systemd-journal/805:
[ 1235.990616] #0: ff1100180e0372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1236.000506] #1: ffffffffb21033a0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1236.013330] 4 locks held by python3/3833:
[ 1236.018626] #0: ff1100012eb5c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.028515] #1: ff11000179ab3e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.039469] #2: ff11000121344b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.050626] #3: ff110001213e9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.060714] 4 locks held by python3/5879:
[ 1236.066012] #0: ff1100012eb5c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.075892] #1: ff110001cc826880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.086851] #2: ff11000121344b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.098005] #3: ff110001213e9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.108097]
[ 1236.110563] =============================================\x0a
[ 1272.272077] kworker/u129:14 (309) used greatest stack depth: 21936 bytes left
[ 1358.289809] INFO: task python3:5879 blocked for more than 491 seconds.
[ 1358.297969] Not tainted 7.1.0-rc3-llwx-gef069d5c7dcc #1
[ 1358.305223] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.314817] task:python3 state:D stack:25672 pid:5879 tgid:5879 ppid:5878 task_flags:0x400100 flags:0x00080002
[ 1358.328025] Call Trace:
[ 1358.331566]
[ 1358.334721] __schedule+0xcc3/0x19c0
[ 1358.339540] ? io_schedule_timeout+0x140/0x140
[ 1358.345326] ? find_held_lock+0x2b/0x80
[ 1358.350437] schedule+0xd4/0x260
[ 1358.354857] schedule_preempt_disabled+0x18/0x30
[ 1358.360828] __mutex_lock+0x6a3/0x2000
[ 1358.365823] ? threaded_store+0x132/0x220
[ 1358.371125] ? ww_mutex_lock+0x160/0x160
[ 1358.376326] ? selinux_inode_free_security+0x330/0x330
[ 1358.382899] ? cap_capable+0x1d6/0x3e0
[ 1358.387906] ? threaded_store+0x132/0x220
[ 1358.393200] threaded_store+0x132/0x220
[ 1358.398304] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.403603] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.408902] ? sysfs_kf_write+0x65/0x160
[ 1358.414096] ? sysfs_file_ops+0x140/0x140
[ 1358.419393] kernfs_fop_write_iter+0x335/0x510
[ 1358.425179] ? kernfs_ops+0xf0/0xf0
[ 1358.429883] new_sync_write+0x333/0x750
[ 1358.434986] ? __lock_acquire+0x508/0xc10
[ 1358.440285] ? new_sync_read+0x740/0x740
[ 1358.445484] ? lock_acquire.part.0+0xbc/0x260
[ 1358.451169] ? ksys_write+0x116/0x250
[ 1358.456080] vfs_write+0x657/0x9b0
[ 1358.460695] ksys_write+0x116/0x250
[ 1358.465407] ? __ia32_sys_read+0xc0/0xc0
[ 1358.470607] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.476197] ? rcu_is_watching+0x15/0xd0
[ 1358.481399] do_syscall_64+0xf3/0x650
[ 1358.486307] ? trace_hardirqs_off+0xd/0x30
[ 1358.491701] ? exc_page_fault+0xda/0xf0
[ 1358.496803] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.503272] RIP: 0033:0x7fd87b04222e
[ 1358.508080] RSP: 002b:00007ffd10435b80 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.517377] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fd87b04222e
[ 1358.526176] RDX: 0000000000000001 RSI: 000056307ec4dcd0 RDI: 0000000000000007
[ 1358.534984] RBP: 00007ffd10435b90 R08: 0000000000000000 R09: 0000000000000000
[ 1358.543783] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fd87aeba390
[ 1358.552591] R13: 000056307ec4dcd0 R14: 0000000000000007 R15: 0000000000000003
[ 1358.561405]
[ 1358.564677] INFO: task python3:5879 is blocked on a mutex likely owned by task python3:3833.
[ 1358.574950] task:python3 state:D stack:25584 pid:3833 tgid:3833 ppid:3832 task_flags:0x400100 flags:0x00080802
[ 1358.588153] Call Trace:
[ 1358.591693]
[ 1358.594844] __schedule+0xcc3/0x19c0
[ 1358.599654] ? io_schedule_timeout+0x140/0x140
[ 1358.605428] ? find_held_lock+0x2b/0x80
[ 1358.610531] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.616713] schedule+0xd4/0x260
[ 1358.621130] schedule_timeout+0x13f/0x290
[ 1358.626430] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.632798] ? trace_contention_end+0xb3/0x1a0
[ 1358.638584] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.644266] ? ww_mutex_lock+0x160/0x160
[ 1358.649464] msleep+0x1b/0x30
[ 1358.653593] napi_set_threaded+0x3a5/0x440
[ 1358.658990] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.664487] netif_set_threaded+0x257/0x400
[ 1358.669981] threaded_store+0x1b8/0x220
[ 1358.675085] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.680382] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.685681] ? sysfs_kf_write+0x65/0x160
[ 1358.690881] ? sysfs_file_ops+0x140/0x140
[ 1358.696176] kernfs_fop_write_iter+0x335/0x510
[ 1358.701961] ? kernfs_ops+0xf0/0xf0
[ 1358.706670] new_sync_write+0x333/0x750
[ 1358.711772] ? __lock_acquire+0x508/0xc10
[ 1358.717071] ? new_sync_read+0x740/0x740
[ 1358.722272] ? lock_acquire.part.0+0xbc/0x260
[ 1358.727958] ? ksys_write+0x116/0x250
[ 1358.732870] vfs_write+0x657/0x9b0
[ 1358.737486] ksys_write+0x116/0x250
[ 1358.742197] ? __ia32_sys_read+0xc0/0xc0
[ 1358.747398] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.752988] ? rcu_is_watching+0x15/0xd0
[ 1358.758189] do_syscall_64+0xf3/0x650
[ 1358.763096] ? trace_hardirqs_off+0xd/0x30
[ 1358.768480] ? exc_page_fault+0xda/0xf0
[ 1358.773580] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.780049] RIP: 0033:0x7feeb537b22e
[ 1358.784857] RSP: 002b:00007ffe3dba00c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.794153] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007feeb537b22e
[ 1358.802961] RDX: 0000000000000001 RSI: 00005609330b1cd0 RDI: 0000000000000007
[ 1358.811761] RBP: 00007ffe3dba00d0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.820568] R10: 0000000000000000 R11: 0000000000000202 R12: 00007feeb51f3390
[ 1358.829379] R13: 00005609330b1cd0 R14: 0000000000000007 R15: 0000000000000003
[ 1358.838191]
[ 1358.841460] \x0aShowing all locks held in the system:
[ 1358.850000] 1 lock held by khungtaskd/185:
[ 1358.855396] #0: ffffffffb1efe060 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.866867] 3 locks held by systemd-journal/805:
[ 1358.872846] #0: ff1100180e3b72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1358.882734] #1: ffffffffb21033a0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1358.895551] #2: ff1100012e03f8e0 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 1358.907109] 4 locks held by python3/3833:
[ 1358.912403] #0: ff1100012eb5c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.922290] #1: ff11000179ab3e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.933255] #2: ff11000121344b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.944412] #3: ff110001213e9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.954498] 4 locks held by python3/5879:
[ 1358.959793] #0: ff1100012eb5c3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.969682] #1: ff110001cc826880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.980650] #2: ff11000121344b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.991805] #3: ff110001213e9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.001897]
[ 1359.004363] =============================================\x0a