[ 784.743444] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 784.745850] nipa-hw-worker: [5/46] Retrying drivers/net:napi_threaded.py
[ 784.902381] kselftest: Running tests in drivers/net
[ 801.271306] kworker/u129:12 (305) used greatest stack depth: 21456 bytes left
[ 989.688140] INFO: task python3:5712 blocked for more than 122 seconds.
[ 989.696369] Not tainted 7.1.0-rc5-ncjm-g7e415839c264 #1
[ 989.704254] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.713843] task:python3 state:D stack:25664 pid:5712 tgid:5712 ppid:5711 task_flags:0x400100 flags:0x00080000
[ 989.727049] Call Trace:
[ 989.730591]
[ 989.733733] __schedule+0xcc3/0x19c0
[ 989.738550] ? io_schedule_timeout+0x140/0x140
[ 989.744334] ? find_held_lock+0x2b/0x80
[ 989.749442] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.755329] schedule+0xd8/0x260
[ 989.759748] schedule_preempt_disabled+0x18/0x30
[ 989.765726] __mutex_lock+0x6af/0x2010
[ 989.770731] ? threaded_store+0x132/0x220
[ 989.776031] ? ww_mutex_lock+0x160/0x160
[ 989.781220] ? selinux_inode_free_security+0x330/0x330
[ 989.787790] ? cap_capable+0x1d6/0x3e0
[ 989.792804] ? threaded_store+0x132/0x220
[ 989.798100] threaded_store+0x132/0x220
[ 989.803204] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.808501] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.813802] ? sysfs_kf_write+0x65/0x160
[ 989.819000] ? sysfs_file_ops+0x140/0x140
[ 989.824296] kernfs_fop_write_iter+0x335/0x510
[ 989.830083] ? kernfs_ops+0xf0/0xf0
[ 989.834793] new_sync_write+0x333/0x750
[ 989.839899] ? __lock_acquire+0x508/0xc10
[ 989.845198] ? new_sync_read+0x740/0x740
[ 989.850400] ? lock_acquire.part.0+0xbc/0x260
[ 989.856086] ? ksys_write+0x116/0x250
[ 989.860998] vfs_write+0x657/0x9b0
[ 989.865613] ksys_write+0x116/0x250
[ 989.870315] ? __ia32_sys_read+0xc0/0xc0
[ 989.875516] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.881097] ? rcu_is_watching+0x15/0xd0
[ 989.886300] do_syscall_64+0xf3/0x650
[ 989.891208] ? trace_hardirqs_off+0xd/0x30
[ 989.896603] ? exc_page_fault+0xda/0xf0
[ 989.901709] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.908176] RIP: 0033:0x7f385913122e
[ 989.912987] RSP: 002b:00007ffd61e859b0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.922288] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f385913122e
[ 989.931096] RDX: 0000000000000001 RSI: 00005650c70396b0 RDI: 0000000000000007
[ 989.939904] RBP: 00007ffd61e859c0 R08: 0000000000000000 R09: 0000000000000000
[ 989.948713] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3858fa9390
[ 989.957519] R13: 00005650c70396b0 R14: 0000000000000007 R15: 0000000000000003
[ 989.966342]
[ 989.969651] INFO: task python3:5712 is blocked on a mutex likely owned by task python3:3653.
[ 989.979914] task:python3 state:D stack:25672 pid:3653 tgid:3653 ppid:3652 task_flags:0x400100 flags:0x00080802
[ 989.993115] Call Trace:
[ 989.996655]
[ 989.999805] __schedule+0xcc3/0x19c0
[ 990.004619] ? io_schedule_timeout+0x140/0x140
[ 990.010402] ? find_held_lock+0x2b/0x80
[ 990.015507] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.021686] schedule+0xd8/0x260
[ 990.026105] schedule_timeout+0x13f/0x290
[ 990.031401] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.037769] ? trace_contention_end+0xb3/0x1a0
[ 990.043556] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.049246] ? ww_mutex_lock+0x160/0x160
[ 990.054449] msleep+0x1b/0x30
[ 990.058573] napi_set_threaded+0x3a5/0x440
[ 990.063973] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.069468] netif_set_threaded+0x257/0x400
[ 990.074968] threaded_store+0x1b8/0x220
[ 990.080071] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.085359] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.090655] ? sysfs_kf_write+0x65/0x160
[ 990.095856] ? sysfs_file_ops+0x140/0x140
[ 990.101154] kernfs_fop_write_iter+0x335/0x510
[ 990.106930] ? kernfs_ops+0xf0/0xf0
[ 990.111634] new_sync_write+0x333/0x750
[ 990.116730] ? __lock_acquire+0x508/0xc10
[ 990.122027] ? new_sync_read+0x740/0x740
[ 990.127227] ? lock_acquire.part.0+0xbc/0x260
[ 990.132911] ? ksys_write+0x116/0x250
[ 990.137821] vfs_write+0x657/0x9b0
[ 990.142435] ksys_write+0x116/0x250
[ 990.147148] ? __ia32_sys_read+0xc0/0xc0
[ 990.152339] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.157928] ? rcu_is_watching+0x15/0xd0
[ 990.163128] do_syscall_64+0xf3/0x650
[ 990.168036] ? irq_exit_rcu+0x1a/0x30
[ 990.172946] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.179413] RIP: 0033:0x7f28a72f522e
[ 990.184221] RSP: 002b:00007ffd6508f430 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.193519] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f28a72f522e
[ 990.202340] RDX: 0000000000000001 RSI: 0000561c718696b0 RDI: 0000000000000007
[ 990.211149] RBP: 00007ffd6508f440 R08: 0000000000000000 R09: 0000000000000000
[ 990.219954] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f28a716d390
[ 990.228762] R13: 0000561c718696b0 R14: 0000000000000007 R15: 0000000000000003
[ 990.237567]
[ 990.240827] \x0aShowing all locks held in the system:
[ 990.249369] 1 lock held by khungtaskd/184:
[ 990.254754] #0: ffffffff8f2fdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.266230] 3 locks held by systemd-journal/808:
[ 990.272208] #0: ff1100180dcb72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 990.282095] #1: ffffffff8f502c20 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 990.294905] #2: ff1100019b0649c0 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 990.306466] 4 locks held by python3/3653:
[ 990.311764] #0: ff1100012fcbc3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.321646] #1: ff1100011c628280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.332610] #2: ff11000122a24b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.343767] #3: ff11000122af9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.353865] 4 locks held by python3/5712:
[ 990.359163] #0: ff1100012fcbc3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.369054] #1: ff110001e720fa80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.380017] #2: ff11000122a24b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.391175] #3: ff11000122af9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.401263]
[ 990.403731] =============================================\x0a
[ 1112.570293] INFO: task python3:5712 blocked for more than 245 seconds.
[ 1112.578441] Not tainted 7.1.0-rc5-ncjm-g7e415839c264 #1
[ 1112.585694] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.595288] task:python3 state:D stack:25664 pid:5712 tgid:5712 ppid:5711 task_flags:0x400100 flags:0x00080000
[ 1112.608493] Call Trace:
[ 1112.612035]
[ 1112.615184] __schedule+0xcc3/0x19c0
[ 1112.620003] ? io_schedule_timeout+0x140/0x140
[ 1112.625790] ? find_held_lock+0x2b/0x80
[ 1112.630895] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.636770] schedule+0xd8/0x260
[ 1112.641188] schedule_preempt_disabled+0x18/0x30
[ 1112.647170] __mutex_lock+0x6af/0x2010
[ 1112.652175] ? threaded_store+0x132/0x220
[ 1112.657475] ? ww_mutex_lock+0x160/0x160
[ 1112.662667] ? selinux_inode_free_security+0x330/0x330
[ 1112.669238] ? cap_capable+0x1d6/0x3e0
[ 1112.674249] ? threaded_store+0x132/0x220
[ 1112.679546] threaded_store+0x132/0x220
[ 1112.684648] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.689939] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.695242] ? sysfs_kf_write+0x65/0x160
[ 1112.700442] ? sysfs_file_ops+0x140/0x140
[ 1112.705739] kernfs_fop_write_iter+0x335/0x510
[ 1112.711514] ? kernfs_ops+0xf0/0xf0
[ 1112.716227] new_sync_write+0x333/0x750
[ 1112.721333] ? __lock_acquire+0x508/0xc10
[ 1112.726630] ? new_sync_read+0x740/0x740
[ 1112.731833] ? lock_acquire.part.0+0xbc/0x260
[ 1112.737519] ? ksys_write+0x116/0x250
[ 1112.742431] vfs_write+0x657/0x9b0
[ 1112.747049] ksys_write+0x116/0x250
[ 1112.751763] ? __ia32_sys_read+0xc0/0xc0
[ 1112.756960] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.762547] ? rcu_is_watching+0x15/0xd0
[ 1112.767748] do_syscall_64+0xf3/0x650
[ 1112.772644] ? trace_hardirqs_off+0xd/0x30
[ 1112.778041] ? exc_page_fault+0xda/0xf0
[ 1112.783144] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.789612] RIP: 0033:0x7f385913122e
[ 1112.794425] RSP: 002b:00007ffd61e859b0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.803724] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f385913122e
[ 1112.812535] RDX: 0000000000000001 RSI: 00005650c70396b0 RDI: 0000000000000007
[ 1112.821341] RBP: 00007ffd61e859c0 R08: 0000000000000000 R09: 0000000000000000
[ 1112.830149] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3858fa9390
[ 1112.838950] R13: 00005650c70396b0 R14: 0000000000000007 R15: 0000000000000003
[ 1112.847767]
[ 1112.851070] INFO: task python3:5712 is blocked on a mutex likely owned by task python3:3653.
[ 1112.861343] task:python3 state:D stack:25672 pid:3653 tgid:3653 ppid:3652 task_flags:0x400100 flags:0x00080802
[ 1112.874548] Call Trace:
[ 1112.878087]
[ 1112.881240] __schedule+0xcc3/0x19c0
[ 1112.886055] ? io_schedule_timeout+0x140/0x140
[ 1112.891839] ? find_held_lock+0x2b/0x80
[ 1112.896946] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.903123] schedule+0xd8/0x260
[ 1112.907543] schedule_timeout+0x13f/0x290
[ 1112.912832] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.919194] ? trace_contention_end+0xb3/0x1a0
[ 1112.924981] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.930674] ? ww_mutex_lock+0x160/0x160
[ 1112.935875] msleep+0x1b/0x30
[ 1112.940002] napi_set_threaded+0x3a5/0x440
[ 1112.945400] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.950894] netif_set_threaded+0x257/0x400
[ 1112.956388] threaded_store+0x1b8/0x220
[ 1112.961491] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.966789] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.972085] ? sysfs_kf_write+0x65/0x160
[ 1112.977286] ? sysfs_file_ops+0x140/0x140
[ 1112.982584] kernfs_fop_write_iter+0x335/0x510
[ 1112.988362] ? kernfs_ops+0xf0/0xf0
[ 1112.993072] new_sync_write+0x333/0x750
[ 1112.998176] ? __lock_acquire+0x508/0xc10
[ 1113.003472] ? new_sync_read+0x740/0x740
[ 1113.008675] ? lock_acquire.part.0+0xbc/0x260
[ 1113.014361] ? ksys_write+0x116/0x250
[ 1113.019271] vfs_write+0x657/0x9b0
[ 1113.023887] ksys_write+0x116/0x250
[ 1113.028601] ? __ia32_sys_read+0xc0/0xc0
[ 1113.033802] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.039389] ? rcu_is_watching+0x15/0xd0
[ 1113.044588] do_syscall_64+0xf3/0x650
[ 1113.049494] ? irq_exit_rcu+0x1a/0x30
[ 1113.054402] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.060872] RIP: 0033:0x7f28a72f522e
[ 1113.065681] RSP: 002b:00007ffd6508f430 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.074969] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f28a72f522e
[ 1113.083779] RDX: 0000000000000001 RSI: 0000561c718696b0 RDI: 0000000000000007
[ 1113.092588] RBP: 00007ffd6508f440 R08: 0000000000000000 R09: 0000000000000000
[ 1113.101396] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f28a716d390
[ 1113.110207] R13: 0000561c718696b0 R14: 0000000000000007 R15: 0000000000000003
[ 1113.119024]
[ 1113.122278] \x0aShowing all locks held in the system:
[ 1113.130822] 1 lock held by khungtaskd/184:
[ 1113.136214] #0: ffffffff8f2fdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.147690] 3 locks held by systemd-journal/808:
[ 1113.153669] #0: ff1100180e2b72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1113.163557] #1: ffffffff8f502c20 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1113.176375] #2: ff1100019b064d30 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 1113.187935] 4 locks held by python3/3653:
[ 1113.193230] #0: ff1100012fcbc3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.203120] #1: ff1100011c628280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.214082] #2: ff11000122a24b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.225239] #3: ff11000122af9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.235345] 4 locks held by python3/5712:
[ 1113.240641] #0: ff1100012fcbc3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.250530] #1: ff110001e720fa80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.261493] #2: ff11000122a24b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.272649] #3: ff11000122af9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.282746]
[ 1113.285225] =============================================\x0a
[ 1235.450474] INFO: task python3:5712 blocked for more than 368 seconds.
[ 1235.458622] Not tainted 7.1.0-rc5-ncjm-g7e415839c264 #1
[ 1235.465871] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.475457] task:python3 state:D stack:25664 pid:5712 tgid:5712 ppid:5711 task_flags:0x400100 flags:0x00080002
[ 1235.488664] Call Trace:
[ 1235.492204]
[ 1235.495352] __schedule+0xcc3/0x19c0
[ 1235.500169] ? io_schedule_timeout+0x140/0x140
[ 1235.505956] ? find_held_lock+0x2b/0x80
[ 1235.511063] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.516945] schedule+0xd8/0x260
[ 1235.521360] schedule_preempt_disabled+0x18/0x30
[ 1235.527337] __mutex_lock+0x6af/0x2010
[ 1235.532342] ? threaded_store+0x132/0x220
[ 1235.537640] ? ww_mutex_lock+0x160/0x160
[ 1235.542838] ? selinux_inode_free_security+0x330/0x330
[ 1235.549407] ? cap_capable+0x1d6/0x3e0
[ 1235.554415] ? threaded_store+0x132/0x220
[ 1235.559713] threaded_store+0x132/0x220
[ 1235.564804] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.570098] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.575398] ? sysfs_kf_write+0x65/0x160
[ 1235.580599] ? sysfs_file_ops+0x140/0x140
[ 1235.585892] kernfs_fop_write_iter+0x335/0x510
[ 1235.591675] ? kernfs_ops+0xf0/0xf0
[ 1235.596385] new_sync_write+0x333/0x750
[ 1235.601485] ? __lock_acquire+0x508/0xc10
[ 1235.606780] ? new_sync_read+0x740/0x740
[ 1235.611982] ? lock_acquire.part.0+0xbc/0x260
[ 1235.617667] ? ksys_write+0x116/0x250
[ 1235.622575] vfs_write+0x657/0x9b0
[ 1235.627188] ksys_write+0x116/0x250
[ 1235.631901] ? __ia32_sys_read+0xc0/0xc0
[ 1235.637097] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.642684] ? rcu_is_watching+0x15/0xd0
[ 1235.647884] do_syscall_64+0xf3/0x650
[ 1235.652788] ? trace_hardirqs_off+0xd/0x30
[ 1235.658179] ? exc_page_fault+0xda/0xf0
[ 1235.663282] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.669738] RIP: 0033:0x7f385913122e
[ 1235.674545] RSP: 002b:00007ffd61e859b0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.683839] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f385913122e
[ 1235.692645] RDX: 0000000000000001 RSI: 00005650c70396b0 RDI: 0000000000000007
[ 1235.701452] RBP: 00007ffd61e859c0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.710258] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3858fa9390
[ 1235.719062] R13: 00005650c70396b0 R14: 0000000000000007 R15: 0000000000000003
[ 1235.727873]
[ 1235.731149] INFO: task python3:5712 is blocked on a mutex likely owned by task python3:3653.
[ 1235.741420] task:python3 state:D stack:25672 pid:3653 tgid:3653 ppid:3652 task_flags:0x400100 flags:0x00080802
[ 1235.754621] Call Trace:
[ 1235.758157]
[ 1235.761309] __schedule+0xcc3/0x19c0
[ 1235.766120] ? io_schedule_timeout+0x140/0x140
[ 1235.771893] ? find_held_lock+0x2b/0x80
[ 1235.776994] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.783163] schedule+0xd8/0x260
[ 1235.787582] schedule_timeout+0x13f/0x290
[ 1235.792871] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.799238] ? trace_contention_end+0xb3/0x1a0
[ 1235.805021] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.810710] ? ww_mutex_lock+0x160/0x160
[ 1235.815908] msleep+0x1b/0x30
[ 1235.820032] napi_set_threaded+0x3a5/0x440
[ 1235.825428] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.830914] netif_set_threaded+0x257/0x400
[ 1235.836406] threaded_store+0x1b8/0x220
[ 1235.841506] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.846802] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.852095] ? sysfs_kf_write+0x65/0x160
[ 1235.857293] ? sysfs_file_ops+0x140/0x140
[ 1235.862589] kernfs_fop_write_iter+0x335/0x510
[ 1235.868374] ? kernfs_ops+0xf0/0xf0
[ 1235.873074] new_sync_write+0x333/0x750
[ 1235.878174] ? __lock_acquire+0x508/0xc10
[ 1235.883468] ? new_sync_read+0x740/0x740
[ 1235.888666] ? lock_acquire.part.0+0xbc/0x260
[ 1235.894353] ? ksys_write+0x116/0x250
[ 1235.899261] vfs_write+0x657/0x9b0
[ 1235.903876] ksys_write+0x116/0x250
[ 1235.908580] ? __ia32_sys_read+0xc0/0xc0
[ 1235.913777] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.919363] ? rcu_is_watching+0x15/0xd0
[ 1235.924560] do_syscall_64+0xf3/0x650
[ 1235.929463] ? irq_exit_rcu+0x1a/0x30
[ 1235.934370] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.940837] RIP: 0033:0x7f28a72f522e
[ 1235.945644] RSP: 002b:00007ffd6508f430 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.954941] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f28a72f522e
[ 1235.963750] RDX: 0000000000000001 RSI: 0000561c718696b0 RDI: 0000000000000007
[ 1235.972545] RBP: 00007ffd6508f440 R08: 0000000000000000 R09: 0000000000000000
[ 1235.981354] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f28a716d390
[ 1235.990161] R13: 0000561c718696b0 R14: 0000000000000007 R15: 0000000000000003
[ 1235.998965]
[ 1236.002227] \x0aShowing all locks held in the system:
[ 1236.010765] 1 lock held by khungtaskd/184:
[ 1236.016157] #0: ffffffff8f2fdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.027627] 2 locks held by systemd-journal/808:
[ 1236.033602] #0: ff1100180e7372e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1236.043489] #1: ffffffff8f502c20 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 1236.056320] 4 locks held by python3/3653:
[ 1236.061605] #0: ff1100012fcbc3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.071493] #1: ff1100011c628280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.082452] #2: ff11000122a24b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.093606] #3: ff11000122af9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.103700] 4 locks held by python3/5712:
[ 1236.108995] #0: ff1100012fcbc3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.118883] #1: ff110001e720fa80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.129841] #2: ff11000122a24b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.140994] #3: ff11000122af9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.151083]
[ 1236.153547] =============================================\x0a
[ 1358.332638] INFO: task python3:5712 blocked for more than 491 seconds.
[ 1358.340786] Not tainted 7.1.0-rc5-ncjm-g7e415839c264 #1
[ 1358.348039] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.357627] task:python3 state:D stack:25664 pid:5712 tgid:5712 ppid:5711 task_flags:0x400100 flags:0x00080002
[ 1358.370833] Call Trace:
[ 1358.374374]
[ 1358.377523] __schedule+0xcc3/0x19c0
[ 1358.382342] ? io_schedule_timeout+0x140/0x140
[ 1358.388129] ? find_held_lock+0x2b/0x80
[ 1358.393237] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.399121] schedule+0xd8/0x260
[ 1358.403539] schedule_preempt_disabled+0x18/0x30
[ 1358.409516] __mutex_lock+0x6af/0x2010
[ 1358.414520] ? threaded_store+0x132/0x220
[ 1358.419819] ? ww_mutex_lock+0x160/0x160
[ 1358.425010] ? selinux_inode_free_security+0x330/0x330
[ 1358.431581] ? cap_capable+0x1d6/0x3e0
[ 1358.436590] ? threaded_store+0x132/0x220
[ 1358.441889] threaded_store+0x132/0x220
[ 1358.446990] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.452289] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.457589] ? sysfs_kf_write+0x65/0x160
[ 1358.462792] ? sysfs_file_ops+0x140/0x140
[ 1358.468090] kernfs_fop_write_iter+0x335/0x510
[ 1358.473873] ? kernfs_ops+0xf0/0xf0
[ 1358.478585] new_sync_write+0x333/0x750
[ 1358.483690] ? __lock_acquire+0x508/0xc10
[ 1358.488977] ? new_sync_read+0x740/0x740
[ 1358.494180] ? lock_acquire.part.0+0xbc/0x260
[ 1358.499867] ? ksys_write+0x116/0x250
[ 1358.504777] vfs_write+0x657/0x9b0
[ 1358.509395] ksys_write+0x116/0x250
[ 1358.514107] ? __ia32_sys_read+0xc0/0xc0
[ 1358.519297] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.524889] ? rcu_is_watching+0x15/0xd0
[ 1358.530090] do_syscall_64+0xf3/0x650
[ 1358.534996] ? trace_hardirqs_off+0xd/0x30
[ 1358.540393] ? exc_page_fault+0xda/0xf0
[ 1358.545496] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.551963] RIP: 0033:0x7f385913122e
[ 1358.556765] RSP: 002b:00007ffd61e859b0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.566064] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f385913122e
[ 1358.574871] RDX: 0000000000000001 RSI: 00005650c70396b0 RDI: 0000000000000007
[ 1358.583681] RBP: 00007ffd61e859c0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.592488] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3858fa9390
[ 1358.601299] R13: 00005650c70396b0 R14: 0000000000000007 R15: 0000000000000003
[ 1358.610114]
[ 1358.613382] INFO: task python3:5712 is blocked on a mutex likely owned by task python3:3653.
[ 1358.623652] task:python3 state:D stack:25672 pid:3653 tgid:3653 ppid:3652 task_flags:0x400100 flags:0x00080802
[ 1358.636856] Call Trace:
[ 1358.640395]
[ 1358.643543] __schedule+0xcc3/0x19c0
[ 1358.648356] ? io_schedule_timeout+0x140/0x140
[ 1358.654142] ? find_held_lock+0x2b/0x80
[ 1358.659247] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.665425] schedule+0xd8/0x260
[ 1358.669844] schedule_timeout+0x13f/0x290
[ 1358.675142] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.681512] ? trace_contention_end+0xb3/0x1a0
[ 1358.687297] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.692989] ? ww_mutex_lock+0x160/0x160
[ 1358.698191] msleep+0x1b/0x30
[ 1358.702317] napi_set_threaded+0x3a5/0x440
[ 1358.707717] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.713211] netif_set_threaded+0x257/0x400
[ 1358.718706] threaded_store+0x1b8/0x220
[ 1358.723808] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.729099] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.734394] ? sysfs_kf_write+0x65/0x160
[ 1358.739595] ? sysfs_file_ops+0x140/0x140
[ 1358.744882] kernfs_fop_write_iter+0x335/0x510
[ 1358.750667] ? kernfs_ops+0xf0/0xf0
[ 1358.755379] new_sync_write+0x333/0x750
[ 1358.760484] ? __lock_acquire+0x508/0xc10
[ 1358.765780] ? new_sync_read+0x740/0x740
[ 1358.770983] ? lock_acquire.part.0+0xbc/0x260
[ 1358.776669] ? ksys_write+0x116/0x250
[ 1358.781580] vfs_write+0x657/0x9b0
[ 1358.786197] ksys_write+0x116/0x250
[ 1358.790911] ? __ia32_sys_read+0xc0/0xc0
[ 1358.796109] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.801700] ? rcu_is_watching+0x15/0xd0
[ 1358.806900] do_syscall_64+0xf3/0x650
[ 1358.811805] ? irq_exit_rcu+0x1a/0x30
[ 1358.816712] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.823181] RIP: 0033:0x7f28a72f522e
[ 1358.827989] RSP: 002b:00007ffd6508f430 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.837287] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f28a72f522e
[ 1358.846096] RDX: 0000000000000001 RSI: 0000561c718696b0 RDI: 0000000000000007
[ 1358.854906] RBP: 00007ffd6508f440 R08: 0000000000000000 R09: 0000000000000000
[ 1358.863715] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f28a716d390
[ 1358.872524] R13: 0000561c718696b0 R14: 0000000000000007 R15: 0000000000000003
[ 1358.881339]
[ 1358.884611] \x0aShowing all locks held in the system:
[ 1358.893149] 1 lock held by khungtaskd/184:
[ 1358.898540] #0: ffffffff8f2fdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.910012] 3 locks held by systemd-journal/808:
[ 1358.915992] #0: ff1100180ddb72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0
[ 1358.925879] #1: ff110001a2520f78 (&mm->mmap_lock){++++}-{4:4}, at: fs_reclaim_acquire+0x7b/0x120
[ 1358.936645] #2: ff1100019b0649c0 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0
[ 1358.948196] 4 locks held by python3/3653:
[ 1358.953493] #0: ff1100012fcbc3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.963384] #1: ff1100011c628280 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.974346] #2: ff11000122a24b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.985501] #3: ff11000122af9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.995596] 4 locks held by python3/5712:
[ 1359.000890] #0: ff1100012fcbc3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1359.010782] #1: ff110001e720fa80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1359.021743] #2: ff11000122a24b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.032900] #3: ff11000122af9d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.042985]
[ 1359.045454] =============================================\x0a