[ 780.753862] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 780.757240] nipa-hw-worker: [5/46] Retrying drivers/net:napi_threaded.py
[ 780.911127] kselftest: Running tests in drivers/net
[ 989.697815] INFO: task python3:5656 blocked for more than 122 seconds.
[ 989.706048] Not tainted 7.1.0-rc2-jcmo-g6bb4fb3b2b96 #1
[ 989.714013] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.723614] task:python3 state:D stack:25672 pid:5656 tgid:5656 ppid:5655 task_flags:0x400100 flags:0x00080000
[ 989.736825] Call Trace:
[ 989.740366]
[ 989.743519] __schedule+0xcba/0x1940
[ 989.748336] ? io_schedule_timeout+0x140/0x140
[ 989.754125] ? find_held_lock+0x2b/0x80
[ 989.759229] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.765114] schedule+0xd4/0x260
[ 989.769533] schedule_preempt_disabled+0x18/0x30
[ 989.775513] __mutex_lock+0x6a3/0x2000
[ 989.780511] ? threaded_store+0x132/0x220
[ 989.785813] ? ww_mutex_lock+0x160/0x160
[ 989.791014] ? selinux_inode_free_security+0x330/0x330
[ 989.797586] ? cap_capable+0x1d6/0x3e0
[ 989.802595] ? threaded_store+0x132/0x220
[ 989.807893] threaded_store+0x132/0x220
[ 989.812996] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.818296] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.823596] ? sysfs_kf_write+0x65/0x160
[ 989.828797] ? sysfs_file_ops+0x140/0x140
[ 989.834094] kernfs_fop_write_iter+0x335/0x510
[ 989.839879] ? kernfs_ops+0xf0/0xf0
[ 989.844592] new_sync_write+0x333/0x750
[ 989.849697] ? __lock_acquire+0x508/0xc10
[ 989.854993] ? new_sync_read+0x740/0x740
[ 989.860196] ? lock_acquire.part.0+0xbc/0x260
[ 989.865885] ? ksys_write+0x116/0x250
[ 989.870797] vfs_write+0x657/0x9b0
[ 989.875414] ksys_write+0x116/0x250
[ 989.880129] ? __ia32_sys_read+0xc0/0xc0
[ 989.885327] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.890919] ? rcu_is_watching+0x15/0xd0
[ 989.896118] do_syscall_64+0xf3/0x650
[ 989.901024] ? trace_hardirqs_off+0xd/0x30
[ 989.906421] ? exc_page_fault+0xda/0xf0
[ 989.911522] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.917993] RIP: 0033:0x7f3da634222e
[ 989.922804] RSP: 002b:00007ffcb8d3acb0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.932105] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3da634222e
[ 989.940916] RDX: 0000000000000001 RSI: 000055d958caefe0 RDI: 0000000000000007
[ 989.949722] RBP: 00007ffcb8d3acc0 R08: 0000000000000000 R09: 0000000000000000
[ 989.958530] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3da61ba390
[ 989.967338] R13: 000055d958caefe0 R14: 0000000000000007 R15: 0000000000000003
[ 989.976152]
[ 989.979460] INFO: task python3:5656 is blocked on a mutex likely owned by task python3:3609.
[ 989.989733] task:python3 state:D stack:25672 pid:3609 tgid:3609 ppid:3608 task_flags:0x400100 flags:0x00080802
[ 990.002937] Call Trace:
[ 990.006476]
[ 990.009626] __schedule+0xcba/0x1940
[ 990.014430] ? io_schedule_timeout+0x140/0x140
[ 990.020213] ? find_held_lock+0x2b/0x80
[ 990.025318] ? __try_to_del_timer_sync+0x155/0x1f0
[ 990.031499] schedule+0xd4/0x260
[ 990.035916] schedule_timeout+0x13f/0x290
[ 990.041213] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.047583] ? trace_contention_end+0xb3/0x1a0
[ 990.053360] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.059043] ? ww_mutex_lock+0x160/0x160
[ 990.064244] msleep+0x1b/0x30
[ 990.068371] napi_set_threaded+0x3a5/0x440
[ 990.073760] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.079255] netif_set_threaded+0x257/0x400
[ 990.084750] threaded_store+0x1b8/0x220
[ 990.089855] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.095153] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.100442] ? sysfs_kf_write+0x65/0x160
[ 990.105642] ? sysfs_file_ops+0x140/0x140
[ 990.110939] kernfs_fop_write_iter+0x335/0x510
[ 990.116718] ? kernfs_ops+0xf0/0xf0
[ 990.121430] new_sync_write+0x333/0x750
[ 990.126531] ? __lock_acquire+0x508/0xc10
[ 990.131828] ? new_sync_read+0x740/0x740
[ 990.137030] ? lock_acquire.part.0+0xbc/0x260
[ 990.142718] ? ksys_write+0x116/0x250
[ 990.147627] vfs_write+0x657/0x9b0
[ 990.152244] ksys_write+0x116/0x250
[ 990.156959] ? __ia32_sys_read+0xc0/0xc0
[ 990.162161] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.167751] ? rcu_is_watching+0x15/0xd0
[ 990.172941] do_syscall_64+0xf3/0x650
[ 990.177845] ? irq_exit_rcu+0x1a/0x30
[ 990.182756] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.189224] RIP: 0033:0x7f6b33d3922e
[ 990.194034] RSP: 002b:00007ffdb1bb8550 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.203333] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f6b33d3922e
[ 990.212141] RDX: 0000000000000001 RSI: 0000561aba035fe0 RDI: 0000000000000007
[ 990.220942] RBP: 00007ffdb1bb8560 R08: 0000000000000000 R09: 0000000000000000
[ 990.229748] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f6b33bb1390
[ 990.238556] R13: 0000561aba035fe0 R14: 0000000000000007 R15: 0000000000000003
[ 990.247370]
[ 990.250630] \x0aShowing all locks held in the system:
[ 990.259151] 2 locks held by systemd/1:
[ 990.264152] #0: ff1100012fbc93e8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.274046] #1: ff110001b6221a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.285031] 1 lock held by khungtaskd/184:
[ 990.290425] #0: ffffffff8acfdf60 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.301896] 2 locks held by systemd-journal/801:
[ 990.307876] #0: ff1100180e4b72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x1940
[ 990.317764] #1: ffffffff8af032a0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120
[ 990.330593] 4 locks held by python3/3609:
[ 990.335891] #0: ff1100012fad93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.345773] #1: ff11000163fa4480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.356739] #2: ff110017da6d8b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.367898] #3: ff110017da769d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.378000] 4 locks held by python3/5656:
[ 990.383298] #0: ff1100012fad93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.393180] #1: ff110001e9bdb880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.404142] #2: ff110017da6d8b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.415299] #3: ff110017da769d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.425386]
[ 990.427853] =============================================\x0a
[ 1036.290275] kworker/u129:0 (177) used greatest stack depth: 21936 bytes left
[ 1112.579202] INFO: task python3:5656 blocked for more than 245 seconds.
[ 1112.587365] Not tainted 7.1.0-rc2-jcmo-g6bb4fb3b2b96 #1
[ 1112.594619] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.604211] task:python3 state:D stack:25672 pid:5656 tgid:5656 ppid:5655 task_flags:0x400100 flags:0x00080000
[ 1112.617420] Call Trace:
[ 1112.620961]
[ 1112.624114] __schedule+0xcba/0x1940
[ 1112.628933] ? io_schedule_timeout+0x140/0x140
[ 1112.634718] ? find_held_lock+0x2b/0x80
[ 1112.639826] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.645711] schedule+0xd4/0x260
[ 1112.650132] schedule_preempt_disabled+0x18/0x30
[ 1112.656113] __mutex_lock+0x6a3/0x2000
[ 1112.661116] ? threaded_store+0x132/0x220
[ 1112.666417] ? ww_mutex_lock+0x160/0x160
[ 1112.671617] ? selinux_inode_free_security+0x330/0x330
[ 1112.678185] ? cap_capable+0x1d6/0x3e0
[ 1112.683194] ? threaded_store+0x132/0x220
[ 1112.688490] threaded_store+0x132/0x220
[ 1112.693594] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.698892] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.704192] ? sysfs_kf_write+0x65/0x160
[ 1112.709391] ? sysfs_file_ops+0x140/0x140
[ 1112.714688] kernfs_fop_write_iter+0x335/0x510
[ 1112.720475] ? kernfs_ops+0xf0/0xf0
[ 1112.725185] new_sync_write+0x333/0x750
[ 1112.730280] ? __lock_acquire+0x508/0xc10
[ 1112.735577] ? new_sync_read+0x740/0x740
[ 1112.740779] ? lock_acquire.part.0+0xbc/0x260
[ 1112.746468] ? ksys_write+0x116/0x250
[ 1112.751381] vfs_write+0x657/0x9b0
[ 1112.755999] ksys_write+0x116/0x250
[ 1112.760712] ? __ia32_sys_read+0xc0/0xc0
[ 1112.765904] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.771494] ? rcu_is_watching+0x15/0xd0
[ 1112.776698] do_syscall_64+0xf3/0x650
[ 1112.781607] ? trace_hardirqs_off+0xd/0x30
[ 1112.787002] ? exc_page_fault+0xda/0xf0
[ 1112.792105] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.798576] RIP: 0033:0x7f3da634222e
[ 1112.803387] RSP: 002b:00007ffcb8d3acb0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.812685] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3da634222e
[ 1112.821495] RDX: 0000000000000001 RSI: 000055d958caefe0 RDI: 0000000000000007
[ 1112.830304] RBP: 00007ffcb8d3acc0 R08: 0000000000000000 R09: 0000000000000000
[ 1112.839113] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3da61ba390
[ 1112.847922] R13: 000055d958caefe0 R14: 0000000000000007 R15: 0000000000000003
[ 1112.856735]
[ 1112.860039] INFO: task python3:5656 is blocked on a mutex likely owned by task python3:3609.
[ 1112.870311] task:python3 state:D stack:25672 pid:3609 tgid:3609 ppid:3608 task_flags:0x400100 flags:0x00080802
[ 1112.883513] Call Trace:
[ 1112.887050]
[ 1112.890201] __schedule+0xcba/0x1940
[ 1112.895003] ? io_schedule_timeout+0x140/0x140
[ 1112.900787] ? find_held_lock+0x2b/0x80
[ 1112.905889] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.912066] schedule+0xd4/0x260
[ 1112.916484] schedule_timeout+0x13f/0x290
[ 1112.921780] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.928149] ? trace_contention_end+0xb3/0x1a0
[ 1112.933935] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.939625] ? ww_mutex_lock+0x160/0x160
[ 1112.944824] msleep+0x1b/0x30
[ 1112.948949] napi_set_threaded+0x3a5/0x440
[ 1112.954347] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.959845] netif_set_threaded+0x257/0x400
[ 1112.965339] threaded_store+0x1b8/0x220
[ 1112.970442] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.975737] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.981036] ? sysfs_kf_write+0x65/0x160
[ 1112.986235] ? sysfs_file_ops+0x140/0x140
[ 1112.991530] kernfs_fop_write_iter+0x335/0x510
[ 1112.997316] ? kernfs_ops+0xf0/0xf0
[ 1113.002017] new_sync_write+0x333/0x750
[ 1113.007119] ? __lock_acquire+0x508/0xc10
[ 1113.012420] ? new_sync_read+0x740/0x740
[ 1113.017620] ? lock_acquire.part.0+0xbc/0x260
[ 1113.023306] ? ksys_write+0x116/0x250
[ 1113.028215] vfs_write+0x657/0x9b0
[ 1113.032831] ksys_write+0x116/0x250
[ 1113.037542] ? __ia32_sys_read+0xc0/0xc0
[ 1113.042743] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.048334] ? rcu_is_watching+0x15/0xd0
[ 1113.053526] do_syscall_64+0xf3/0x650
[ 1113.058434] ? irq_exit_rcu+0x1a/0x30
[ 1113.063343] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.069810] RIP: 0033:0x7f6b33d3922e
[ 1113.074609] RSP: 002b:00007ffdb1bb8550 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.083905] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f6b33d3922e
[ 1113.092720] RDX: 0000000000000001 RSI: 0000561aba035fe0 RDI: 0000000000000007
[ 1113.101528] RBP: 00007ffdb1bb8560 R08: 0000000000000000 R09: 0000000000000000
[ 1113.110339] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f6b33bb1390
[ 1113.119147] R13: 0000561aba035fe0 R14: 0000000000000007 R15: 0000000000000003
[ 1113.127960]
[ 1113.131222] \x0aShowing all locks held in the system:
[ 1113.139761] 1 lock held by khungtaskd/184:
[ 1113.145155] #0: ffffffff8acfdf60 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.156639] 4 locks held by python3/3609:
[ 1113.161924] #0: ff1100012fad93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.171813] #1: ff11000163fa4480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.182777] #2: ff110017da6d8b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.193936] #3: ff110017da769d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.204037] 4 locks held by python3/5656:
[ 1113.209330] #0: ff1100012fad93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.219219] #1: ff110001e9bdb880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.230183] #2: ff110017da6d8b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.241343] #3: ff110017da769d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.251432]
[ 1113.253898] =============================================\x0a
[ 1235.460554] INFO: task python3:5656 blocked for more than 368 seconds.
[ 1235.468702] Not tainted 7.1.0-rc2-jcmo-g6bb4fb3b2b96 #1
[ 1235.475954] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.485544] task:python3 state:D stack:25672 pid:5656 tgid:5656 ppid:5655 task_flags:0x400100 flags:0x00080002
[ 1235.498750] Call Trace:
[ 1235.502292]
[ 1235.505442] __schedule+0xcba/0x1940
[ 1235.510258] ? io_schedule_timeout+0x140/0x140
[ 1235.516045] ? find_held_lock+0x2b/0x80
[ 1235.521150] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.527036] schedule+0xd4/0x260
[ 1235.531455] schedule_preempt_disabled+0x18/0x30
[ 1235.537434] __mutex_lock+0x6a3/0x2000
[ 1235.542438] ? threaded_store+0x132/0x220
[ 1235.547729] ? ww_mutex_lock+0x160/0x160
[ 1235.552930] ? selinux_inode_free_security+0x330/0x330
[ 1235.559500] ? cap_capable+0x1d6/0x3e0
[ 1235.564511] ? threaded_store+0x132/0x220
[ 1235.569805] threaded_store+0x132/0x220
[ 1235.574907] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.580200] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.585500] ? sysfs_kf_write+0x65/0x160
[ 1235.590703] ? sysfs_file_ops+0x140/0x140
[ 1235.596002] kernfs_fop_write_iter+0x335/0x510
[ 1235.601789] ? kernfs_ops+0xf0/0xf0
[ 1235.606501] new_sync_write+0x333/0x750
[ 1235.611606] ? __lock_acquire+0x508/0xc10
[ 1235.616904] ? new_sync_read+0x740/0x740
[ 1235.622107] ? lock_acquire.part.0+0xbc/0x260
[ 1235.627794] ? ksys_write+0x116/0x250
[ 1235.632696] vfs_write+0x657/0x9b0
[ 1235.637310] ksys_write+0x116/0x250
[ 1235.642024] ? __ia32_sys_read+0xc0/0xc0
[ 1235.647223] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.652813] ? rcu_is_watching+0x15/0xd0
[ 1235.658015] do_syscall_64+0xf3/0x650
[ 1235.662924] ? trace_hardirqs_off+0xd/0x30
[ 1235.668320] ? exc_page_fault+0xda/0xf0
[ 1235.673413] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.679884] RIP: 0033:0x7f3da634222e
[ 1235.684695] RSP: 002b:00007ffcb8d3acb0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.693995] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3da634222e
[ 1235.702806] RDX: 0000000000000001 RSI: 000055d958caefe0 RDI: 0000000000000007
[ 1235.711612] RBP: 00007ffcb8d3acc0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.720419] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3da61ba390
[ 1235.729230] R13: 000055d958caefe0 R14: 0000000000000007 R15: 0000000000000003
[ 1235.738044]
[ 1235.741332] INFO: task python3:5656 is blocked on a mutex likely owned by task python3:3609.
[ 1235.751604] task:python3 state:D stack:25672 pid:3609 tgid:3609 ppid:3608 task_flags:0x400100 flags:0x00080802
[ 1235.764809] Call Trace:
[ 1235.768347]
[ 1235.771495] __schedule+0xcba/0x1940
[ 1235.776307] ? io_schedule_timeout+0x140/0x140
[ 1235.782092] ? find_held_lock+0x2b/0x80
[ 1235.787198] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.793375] schedule+0xd4/0x260
[ 1235.797795] schedule_timeout+0x13f/0x290
[ 1235.803093] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.809463] ? trace_contention_end+0xb3/0x1a0
[ 1235.815251] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.820942] ? ww_mutex_lock+0x160/0x160
[ 1235.826144] msleep+0x1b/0x30
[ 1235.830271] napi_set_threaded+0x3a5/0x440
[ 1235.835669] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.841162] netif_set_threaded+0x257/0x400
[ 1235.846657] threaded_store+0x1b8/0x220
[ 1235.851762] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.857061] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.862360] ? sysfs_kf_write+0x65/0x160
[ 1235.867560] ? sysfs_file_ops+0x140/0x140
[ 1235.872847] kernfs_fop_write_iter+0x335/0x510
[ 1235.878634] ? kernfs_ops+0xf0/0xf0
[ 1235.883344] new_sync_write+0x333/0x750
[ 1235.888448] ? __lock_acquire+0x508/0xc10
[ 1235.893745] ? new_sync_read+0x740/0x740
[ 1235.898947] ? lock_acquire.part.0+0xbc/0x260
[ 1235.904633] ? ksys_write+0x116/0x250
[ 1235.909543] vfs_write+0x657/0x9b0
[ 1235.914160] ksys_write+0x116/0x250
[ 1235.918874] ? __ia32_sys_read+0xc0/0xc0
[ 1235.924072] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.929661] ? rcu_is_watching+0x15/0xd0
[ 1235.934855] do_syscall_64+0xf3/0x650
[ 1235.939764] ? irq_exit_rcu+0x1a/0x30
[ 1235.944671] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.951140] RIP: 0033:0x7f6b33d3922e
[ 1235.955949] RSP: 002b:00007ffdb1bb8550 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.965247] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f6b33d3922e
[ 1235.974056] RDX: 0000000000000001 RSI: 0000561aba035fe0 RDI: 0000000000000007
[ 1235.982867] RBP: 00007ffdb1bb8560 R08: 0000000000000000 R09: 0000000000000000
[ 1235.991675] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f6b33bb1390
[ 1236.000481] R13: 0000561aba035fe0 R14: 0000000000000007 R15: 0000000000000003
[ 1236.009295]
[ 1236.012562] \x0aShowing all locks held in the system:
[ 1236.021102] 1 lock held by khungtaskd/184:
[ 1236.026498] #0: ffffffff8acfdf60 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.037981] 4 locks held by python3/3609:
[ 1236.043276] #0: ff1100012fad93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.053168] #1: ff11000163fa4480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.064132] #2: ff110017da6d8b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.075288] #3: ff110017da769d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.085384] 4 locks held by python3/5656:
[ 1236.090681] #0: ff1100012fad93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.100571] #1: ff110001e9bdb880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.111534] #2: ff110017da6d8b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.122691] #3: ff110017da769d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.132775]
[ 1236.135243] =============================================\x0a
[ 1358.341923] INFO: task python3:5656 blocked for more than 491 seconds.
[ 1358.350073] Not tainted 7.1.0-rc2-jcmo-g6bb4fb3b2b96 #1
[ 1358.357324] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.366914] task:python3 state:D stack:25672 pid:5656 tgid:5656 ppid:5655 task_flags:0x400100 flags:0x00080002
[ 1358.380119] Call Trace:
[ 1358.383660]
[ 1358.386814] __schedule+0xcba/0x1940
[ 1358.391633] ? io_schedule_timeout+0x140/0x140
[ 1358.397420] ? find_held_lock+0x2b/0x80
[ 1358.402529] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.408413] schedule+0xd4/0x260
[ 1358.412830] schedule_preempt_disabled+0x18/0x30
[ 1358.418808] __mutex_lock+0x6a3/0x2000
[ 1358.423812] ? threaded_store+0x132/0x220
[ 1358.429101] ? ww_mutex_lock+0x160/0x160
[ 1358.434302] ? selinux_inode_free_security+0x330/0x330
[ 1358.440873] ? cap_capable+0x1d6/0x3e0
[ 1358.445882] ? threaded_store+0x132/0x220
[ 1358.451180] threaded_store+0x132/0x220
[ 1358.456273] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.461571] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.466871] ? sysfs_kf_write+0x65/0x160
[ 1358.472074] ? sysfs_file_ops+0x140/0x140
[ 1358.477369] kernfs_fop_write_iter+0x335/0x510
[ 1358.483152] ? kernfs_ops+0xf0/0xf0
[ 1358.487865] new_sync_write+0x333/0x750
[ 1358.492967] ? __lock_acquire+0x508/0xc10
[ 1358.498264] ? new_sync_read+0x740/0x740
[ 1358.503467] ? lock_acquire.part.0+0xbc/0x260
[ 1358.509156] ? ksys_write+0x116/0x250
[ 1358.514065] vfs_write+0x657/0x9b0
[ 1358.518680] ksys_write+0x116/0x250
[ 1358.523394] ? __ia32_sys_read+0xc0/0xc0
[ 1358.528592] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.534184] ? rcu_is_watching+0x15/0xd0
[ 1358.539386] do_syscall_64+0xf3/0x650
[ 1358.544291] ? trace_hardirqs_off+0xd/0x30
[ 1358.549688] ? exc_page_fault+0xda/0xf0
[ 1358.554781] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.561243] RIP: 0033:0x7f3da634222e
[ 1358.566055] RSP: 002b:00007ffcb8d3acb0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.575356] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3da634222e
[ 1358.584166] RDX: 0000000000000001 RSI: 000055d958caefe0 RDI: 0000000000000007
[ 1358.592975] RBP: 00007ffcb8d3acc0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.601786] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3da61ba390
[ 1358.610594] R13: 000055d958caefe0 R14: 0000000000000007 R15: 0000000000000003
[ 1358.619399]
[ 1358.622678] INFO: task python3:5656 is blocked on a mutex likely owned by task python3:3609.
[ 1358.632953] task:python3 state:D stack:25672 pid:3609 tgid:3609 ppid:3608 task_flags:0x400100 flags:0x00080802
[ 1358.646155] Call Trace:
[ 1358.649693]
[ 1358.652842] __schedule+0xcba/0x1940
[ 1358.657655] ? io_schedule_timeout+0x140/0x140
[ 1358.663441] ? find_held_lock+0x2b/0x80
[ 1358.668537] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.674715] schedule+0xd4/0x260
[ 1358.679134] schedule_timeout+0x13f/0x290
[ 1358.684434] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.690804] ? trace_contention_end+0xb3/0x1a0
[ 1358.696589] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.702281] ? ww_mutex_lock+0x160/0x160
[ 1358.707483] msleep+0x1b/0x30
[ 1358.711610] napi_set_threaded+0x3a5/0x440
[ 1358.717005] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.722499] netif_set_threaded+0x257/0x400
[ 1358.727995] threaded_store+0x1b8/0x220
[ 1358.733097] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.738395] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.743694] ? sysfs_kf_write+0x65/0x160
[ 1358.748896] ? sysfs_file_ops+0x140/0x140
[ 1358.754193] kernfs_fop_write_iter+0x335/0x510
[ 1358.759967] ? kernfs_ops+0xf0/0xf0
[ 1358.764678] new_sync_write+0x333/0x750
[ 1358.769774] ? __lock_acquire+0x508/0xc10
[ 1358.775072] ? new_sync_read+0x740/0x740
[ 1358.780273] ? lock_acquire.part.0+0xbc/0x260
[ 1358.785961] ? ksys_write+0x116/0x250
[ 1358.790871] vfs_write+0x657/0x9b0
[ 1358.795489] ksys_write+0x116/0x250
[ 1358.800204] ? __ia32_sys_read+0xc0/0xc0
[ 1358.805406] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.810997] ? rcu_is_watching+0x15/0xd0
[ 1358.816196] do_syscall_64+0xf3/0x650
[ 1358.821106] ? irq_exit_rcu+0x1a/0x30
[ 1358.826016] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.832477] RIP: 0033:0x7f6b33d3922e
[ 1358.837285] RSP: 002b:00007ffdb1bb8550 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.846583] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f6b33d3922e
[ 1358.855391] RDX: 0000000000000001 RSI: 0000561aba035fe0 RDI: 0000000000000007
[ 1358.864201] RBP: 00007ffdb1bb8560 R08: 0000000000000000 R09: 0000000000000000
[ 1358.873007] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f6b33bb1390
[ 1358.881817] R13: 0000561aba035fe0 R14: 0000000000000007 R15: 0000000000000003
[ 1358.890632]
[ 1358.893901] \x0aShowing all locks held in the system:
[ 1358.902439] 1 lock held by khungtaskd/184:
[ 1358.907836] #0: ffffffff8acfdf60 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.919320] 4 locks held by python3/3609:
[ 1358.924617] #0: ff1100012fad93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.934509] #1: ff11000163fa4480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.945469] #2: ff110017da6d8b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.956629] #3: ff110017da769d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.966722] 4 locks held by python3/5656:
[ 1358.972018] #0: ff1100012fad93e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.981909] #1: ff110001e9bdb880 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.992870] #2: ff110017da6d8b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1359.004026] #3: ff110017da769d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1359.014119]
[ 1359.016586] =============================================\x0a