[ 783.227006] nipa-hw-worker: drivers/net:napi_threaded.py: timed out [ 783.229442] nipa-hw-worker: [5/46] Retrying drivers/net:napi_threaded.py [ 783.382973] kselftest: Running tests in drivers/net [ 989.709367] INFO: task python3:5740 blocked for more than 122 seconds. [ 989.717605] Not tainted 7.1.0-rc4-ytun-g8252f1aece1a #1 [ 989.725531] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 989.735122] task:python3 state:D stack:25672 pid:5740 tgid:5740 ppid:5739 task_flags:0x400100 flags:0x00080000 [ 989.748333] Call Trace: [ 989.751874] [ 989.755022] __schedule+0xcc3/0x19c0 [ 989.759837] ? io_schedule_timeout+0x140/0x140 [ 989.765622] ? find_held_lock+0x2b/0x80 [ 989.770728] ? __lock_release.isra.0+0x6b/0x1a0 [ 989.776612] schedule+0xd8/0x260 [ 989.781029] schedule_preempt_disabled+0x18/0x30 [ 989.787007] __mutex_lock+0x6af/0x2010 [ 989.792011] ? threaded_store+0x132/0x220 [ 989.797311] ? ww_mutex_lock+0x160/0x160 [ 989.802510] ? selinux_inode_free_security+0x330/0x330 [ 989.809078] ? cap_capable+0x1d6/0x3e0 [ 989.814089] ? threaded_store+0x132/0x220 [ 989.819388] threaded_store+0x132/0x220 [ 989.824488] ? xps_rxqs_store+0x2e0/0x2e0 [ 989.829785] ? sysfs_file_kobj+0xbd/0x1e0 [ 989.835081] ? sysfs_kf_write+0x65/0x160 [ 989.840281] ? sysfs_file_ops+0x140/0x140 [ 989.845566] kernfs_fop_write_iter+0x335/0x510 [ 989.851349] ? kernfs_ops+0xf0/0xf0 [ 989.856059] new_sync_write+0x333/0x750 [ 989.861161] ? __lock_acquire+0x508/0xc10 [ 989.866458] ? new_sync_read+0x740/0x740 [ 989.871655] ? lock_acquire.part.0+0xbc/0x260 [ 989.877341] ? ksys_write+0x116/0x250 [ 989.882250] vfs_write+0x657/0x9b0 [ 989.886865] ksys_write+0x116/0x250 [ 989.891575] ? __ia32_sys_read+0xc0/0xc0 [ 989.896773] ? kernfs_fop_llseek+0x160/0x1f0 [ 989.902359] ? rcu_is_watching+0x15/0xd0 [ 989.907560] do_syscall_64+0xf3/0x650 [ 989.912466] ? trace_hardirqs_off+0xd/0x30 [ 989.917862] ? exc_page_fault+0xda/0xf0 [ 989.922956] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 989.929421] RIP: 0033:0x7f337193822e [ 989.934232] RSP: 002b:00007fffcd73a940 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 989.943529] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f337193822e [ 989.952337] RDX: 0000000000000001 RSI: 00005644224086b0 RDI: 0000000000000007 [ 989.961142] RBP: 00007fffcd73a950 R08: 0000000000000000 R09: 0000000000000000 [ 989.969950] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f33717b0390 [ 989.978760] R13: 00005644224086b0 R14: 0000000000000007 R15: 0000000000000003 [ 989.987579] [ 989.990870] INFO: task python3:5740 is blocked on a mutex likely owned by task python3:3710. [ 990.001144] task:python3 state:D stack:25672 pid:3710 tgid:3710 ppid:3709 task_flags:0x400100 flags:0x00080802 [ 990.014344] Call Trace: [ 990.017874] [ 990.021021] __schedule+0xcc3/0x19c0 [ 990.025831] ? io_schedule_timeout+0x140/0x140 [ 990.031613] ? find_held_lock+0x2b/0x80 [ 990.036712] ? __try_to_del_timer_sync+0x155/0x1f0 [ 990.042890] schedule+0xd8/0x260 [ 990.047306] schedule_timeout+0x13f/0x290 [ 990.052603] ? hrtimer_nanosleep_restart+0x270/0x270 [ 990.058972] ? trace_contention_end+0xb3/0x1a0 [ 990.064756] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 990.070444] ? ww_mutex_lock+0x160/0x160 [ 990.075632] msleep+0x1b/0x30 [ 990.079755] napi_set_threaded+0x3a5/0x440 [ 990.085150] ? napi_resume_irqs+0x1a0/0x1a0 [ 990.090645] netif_set_threaded+0x257/0x400 [ 990.096135] threaded_store+0x1b8/0x220 [ 990.101236] ? xps_rxqs_store+0x2e0/0x2e0 [ 990.106533] ? sysfs_file_kobj+0xbd/0x1e0 [ 990.111828] ? sysfs_kf_write+0x65/0x160 [ 990.117026] ? sysfs_file_ops+0x140/0x140 [ 990.122320] kernfs_fop_write_iter+0x335/0x510 [ 990.128103] ? kernfs_ops+0xf0/0xf0 [ 990.132811] new_sync_write+0x333/0x750 [ 990.137913] ? __lock_acquire+0x508/0xc10 [ 990.143213] ? new_sync_read+0x740/0x740 [ 990.148414] ? lock_acquire.part.0+0xbc/0x260 [ 990.154098] ? ksys_write+0x116/0x250 [ 990.159006] vfs_write+0x657/0x9b0 [ 990.163619] ksys_write+0x116/0x250 [ 990.168329] ? __ia32_sys_read+0xc0/0xc0 [ 990.173530] ? kernfs_fop_llseek+0x160/0x1f0 [ 990.179109] ? rcu_is_watching+0x15/0xd0 [ 990.184308] do_syscall_64+0xf3/0x650 [ 990.189216] ? irq_exit_rcu+0x1a/0x30 [ 990.194120] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 990.200587] RIP: 0033:0x7f88d3a3722e [ 990.205394] RSP: 002b:00007ffede7a1c70 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 990.214690] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f88d3a3722e [ 990.223488] RDX: 0000000000000001 RSI: 000055e8626f86b0 RDI: 0000000000000007 [ 990.232294] RBP: 00007ffede7a1c80 R08: 0000000000000000 R09: 0000000000000000 [ 990.241103] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f88d38af390 [ 990.249908] R13: 000055e8626f86b0 R14: 0000000000000007 R15: 0000000000000003 [ 990.258722] [ 990.261983] \x0aShowing all locks held in the system: [ 990.270520] 1 lock held by khungtaskd/186: [ 990.275914] #0: ffffffffa0efdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 990.287394] 4 locks held by python3/3710: [ 990.292688] #0: ff1100012e7e13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.302575] #1: ff11000162a4be80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.313538] #2: ff11000120e02b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.324694] #3: ff11000120e69d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.334789] 4 locks held by python3/5740: [ 990.340082] #0: ff1100012e7e13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 990.349972] #1: ff1100014fc12c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 990.360934] #2: ff11000120e02b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 990.372090] #3: ff11000120e69d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 990.382178] [ 990.384645] =============================================\x0a [ 1112.590386] INFO: task python3:5740 blocked for more than 245 seconds. [ 1112.598533] Not tainted 7.1.0-rc4-ytun-g8252f1aece1a #1 [ 1112.605781] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1112.615363] task:python3 state:D stack:25672 pid:5740 tgid:5740 ppid:5739 task_flags:0x400100 flags:0x00080000 [ 1112.628570] Call Trace: [ 1112.632112] [ 1112.635271] __schedule+0xcc3/0x19c0 [ 1112.640089] ? io_schedule_timeout+0x140/0x140 [ 1112.645874] ? find_held_lock+0x2b/0x80 [ 1112.650979] ? __lock_release.isra.0+0x6b/0x1a0 [ 1112.656854] schedule+0xd8/0x260 [ 1112.661271] schedule_preempt_disabled+0x18/0x30 [ 1112.667253] __mutex_lock+0x6af/0x2010 [ 1112.672257] ? threaded_store+0x132/0x220 [ 1112.677556] ? ww_mutex_lock+0x160/0x160 [ 1112.682749] ? selinux_inode_free_security+0x330/0x330 [ 1112.689321] ? cap_capable+0x1d6/0x3e0 [ 1112.694331] ? threaded_store+0x132/0x220 [ 1112.699626] threaded_store+0x132/0x220 [ 1112.704728] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.710026] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.715325] ? sysfs_kf_write+0x65/0x160 [ 1112.720519] ? sysfs_file_ops+0x140/0x140 [ 1112.725817] kernfs_fop_write_iter+0x335/0x510 [ 1112.731604] ? kernfs_ops+0xf0/0xf0 [ 1112.736317] new_sync_write+0x333/0x750 [ 1112.741421] ? __lock_acquire+0x508/0xc10 [ 1112.746718] ? new_sync_read+0x740/0x740 [ 1112.751919] ? lock_acquire.part.0+0xbc/0x260 [ 1112.757604] ? ksys_write+0x116/0x250 [ 1112.762514] vfs_write+0x657/0x9b0 [ 1112.767128] ksys_write+0x116/0x250 [ 1112.771839] ? __ia32_sys_read+0xc0/0xc0 [ 1112.777036] ? kernfs_fop_llseek+0x160/0x1f0 [ 1112.782625] ? rcu_is_watching+0x15/0xd0 [ 1112.787820] do_syscall_64+0xf3/0x650 [ 1112.792729] ? trace_hardirqs_off+0xd/0x30 [ 1112.798125] ? exc_page_fault+0xda/0xf0 [ 1112.803219] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1112.809689] RIP: 0033:0x7f337193822e [ 1112.814498] RSP: 002b:00007fffcd73a940 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1112.823796] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f337193822e [ 1112.832606] RDX: 0000000000000001 RSI: 00005644224086b0 RDI: 0000000000000007 [ 1112.841415] RBP: 00007fffcd73a950 R08: 0000000000000000 R09: 0000000000000000 [ 1112.850232] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f33717b0390 [ 1112.859033] R13: 00005644224086b0 R14: 0000000000000007 R15: 0000000000000003 [ 1112.867845] [ 1112.871123] INFO: task python3:5740 is blocked on a mutex likely owned by task python3:3710. [ 1112.881397] task:python3 state:D stack:25672 pid:3710 tgid:3710 ppid:3709 task_flags:0x400100 flags:0x00080802 [ 1112.894601] Call Trace: [ 1112.898139] [ 1112.901289] __schedule+0xcc3/0x19c0 [ 1112.906102] ? io_schedule_timeout+0x140/0x140 [ 1112.911886] ? find_held_lock+0x2b/0x80 [ 1112.916980] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1112.923157] schedule+0xd8/0x260 [ 1112.927576] schedule_timeout+0x13f/0x290 [ 1112.932876] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1112.939245] ? trace_contention_end+0xb3/0x1a0 [ 1112.945033] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1112.950724] ? ww_mutex_lock+0x160/0x160 [ 1112.955925] msleep+0x1b/0x30 [ 1112.960049] napi_set_threaded+0x3a5/0x440 [ 1112.965443] ? napi_resume_irqs+0x1a0/0x1a0 [ 1112.970937] netif_set_threaded+0x257/0x400 [ 1112.976429] threaded_store+0x1b8/0x220 [ 1112.981523] ? xps_rxqs_store+0x2e0/0x2e0 [ 1112.986822] ? sysfs_file_kobj+0xbd/0x1e0 [ 1112.992120] ? sysfs_kf_write+0x65/0x160 [ 1112.997309] ? sysfs_file_ops+0x140/0x140 [ 1113.002605] kernfs_fop_write_iter+0x335/0x510 [ 1113.008389] ? kernfs_ops+0xf0/0xf0 [ 1113.013101] new_sync_write+0x333/0x750 [ 1113.018202] ? __lock_acquire+0x508/0xc10 [ 1113.023499] ? new_sync_read+0x740/0x740 [ 1113.028700] ? lock_acquire.part.0+0xbc/0x260 [ 1113.034386] ? ksys_write+0x116/0x250 [ 1113.039299] vfs_write+0x657/0x9b0 [ 1113.043916] ksys_write+0x116/0x250 [ 1113.048631] ? __ia32_sys_read+0xc0/0xc0 [ 1113.053828] ? kernfs_fop_llseek+0x160/0x1f0 [ 1113.059418] ? rcu_is_watching+0x15/0xd0 [ 1113.064620] do_syscall_64+0xf3/0x650 [ 1113.069528] ? irq_exit_rcu+0x1a/0x30 [ 1113.074438] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1113.080906] RIP: 0033:0x7f88d3a3722e [ 1113.085715] RSP: 002b:00007ffede7a1c70 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1113.095013] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f88d3a3722e [ 1113.103821] RDX: 0000000000000001 RSI: 000055e8626f86b0 RDI: 0000000000000007 [ 1113.112631] RBP: 00007ffede7a1c80 R08: 0000000000000000 R09: 0000000000000000 [ 1113.121439] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f88d38af390 [ 1113.130241] R13: 000055e8626f86b0 R14: 0000000000000007 R15: 0000000000000003 [ 1113.139055] [ 1113.142319] \x0aShowing all locks held in the system: [ 1113.150857] 1 lock held by khungtaskd/186: [ 1113.156251] #0: ffffffffa0efdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1113.167724] 3 locks held by systemd-journal/803: [ 1113.173704] #0: ff1100180e3b72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0 [ 1113.183591] #1: ffffffffa11029a0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120 [ 1113.196399] #2: ff11000189de1200 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0 [ 1113.207960] 4 locks held by python3/3710: [ 1113.213255] #0: ff1100012e7e13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.223146] #1: ff11000162a4be80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.234110] #2: ff11000120e02b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.245269] #3: ff11000120e69d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.255365] 4 locks held by python3/5740: [ 1113.260661] #0: ff1100012e7e13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1113.270550] #1: ff1100014fc12c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1113.281502] #2: ff11000120e02b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1113.292661] #3: ff11000120e69d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1113.302753] [ 1113.305222] =============================================\x0a [ 1235.471455] INFO: task python3:5740 blocked for more than 368 seconds. [ 1235.479602] Not tainted 7.1.0-rc4-ytun-g8252f1aece1a #1 [ 1235.486856] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1235.496445] task:python3 state:D stack:25672 pid:5740 tgid:5740 ppid:5739 task_flags:0x400100 flags:0x00080002 [ 1235.509641] Call Trace: [ 1235.513179] [ 1235.516328] __schedule+0xcc3/0x19c0 [ 1235.521143] ? io_schedule_timeout+0x140/0x140 [ 1235.526927] ? find_held_lock+0x2b/0x80 [ 1235.532023] ? __lock_release.isra.0+0x6b/0x1a0 [ 1235.537910] schedule+0xd8/0x260 [ 1235.542326] schedule_preempt_disabled+0x18/0x30 [ 1235.548304] __mutex_lock+0x6af/0x2010 [ 1235.553308] ? threaded_store+0x132/0x220 [ 1235.558606] ? ww_mutex_lock+0x160/0x160 [ 1235.563802] ? selinux_inode_free_security+0x330/0x330 [ 1235.570370] ? cap_capable+0x1d6/0x3e0 [ 1235.575381] ? threaded_store+0x132/0x220 [ 1235.580677] threaded_store+0x132/0x220 [ 1235.585776] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.591071] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.596367] ? sysfs_kf_write+0x65/0x160 [ 1235.601565] ? sysfs_file_ops+0x140/0x140 [ 1235.606860] kernfs_fop_write_iter+0x335/0x510 [ 1235.612642] ? kernfs_ops+0xf0/0xf0 [ 1235.617351] new_sync_write+0x333/0x750 [ 1235.622453] ? __lock_acquire+0x508/0xc10 [ 1235.627749] ? new_sync_read+0x740/0x740 [ 1235.632948] ? lock_acquire.part.0+0xbc/0x260 [ 1235.638633] ? ksys_write+0x116/0x250 [ 1235.643542] vfs_write+0x657/0x9b0 [ 1235.648157] ksys_write+0x116/0x250 [ 1235.652861] ? __ia32_sys_read+0xc0/0xc0 [ 1235.658052] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.663641] ? rcu_is_watching+0x15/0xd0 [ 1235.668844] do_syscall_64+0xf3/0x650 [ 1235.673750] ? trace_hardirqs_off+0xd/0x30 [ 1235.679137] ? exc_page_fault+0xda/0xf0 [ 1235.684239] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.690700] RIP: 0033:0x7f337193822e [ 1235.695510] RSP: 002b:00007fffcd73a940 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.704807] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f337193822e [ 1235.713608] RDX: 0000000000000001 RSI: 00005644224086b0 RDI: 0000000000000007 [ 1235.722413] RBP: 00007fffcd73a950 R08: 0000000000000000 R09: 0000000000000000 [ 1235.731219] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f33717b0390 [ 1235.740026] R13: 00005644224086b0 R14: 0000000000000007 R15: 0000000000000003 [ 1235.748832] [ 1235.752125] INFO: task python3:5740 is blocked on a mutex likely owned by task python3:3710. [ 1235.762395] task:python3 state:D stack:25672 pid:3710 tgid:3710 ppid:3709 task_flags:0x400100 flags:0x00080802 [ 1235.775597] Call Trace: [ 1235.779135] [ 1235.782282] __schedule+0xcc3/0x19c0 [ 1235.787094] ? io_schedule_timeout+0x140/0x140 [ 1235.792876] ? find_held_lock+0x2b/0x80 [ 1235.797980] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1235.804158] schedule+0xd8/0x260 [ 1235.808573] schedule_timeout+0x13f/0x290 [ 1235.813869] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1235.820238] ? trace_contention_end+0xb3/0x1a0 [ 1235.826023] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1235.831703] ? ww_mutex_lock+0x160/0x160 [ 1235.836896] msleep+0x1b/0x30 [ 1235.841019] napi_set_threaded+0x3a5/0x440 [ 1235.846413] ? napi_resume_irqs+0x1a0/0x1a0 [ 1235.851907] netif_set_threaded+0x257/0x400 [ 1235.857396] threaded_store+0x1b8/0x220 [ 1235.862497] ? xps_rxqs_store+0x2e0/0x2e0 [ 1235.867796] ? sysfs_file_kobj+0xbd/0x1e0 [ 1235.873091] ? sysfs_kf_write+0x65/0x160 [ 1235.878287] ? sysfs_file_ops+0x140/0x140 [ 1235.883580] kernfs_fop_write_iter+0x335/0x510 [ 1235.889363] ? kernfs_ops+0xf0/0xf0 [ 1235.894073] new_sync_write+0x333/0x750 [ 1235.899172] ? __lock_acquire+0x508/0xc10 [ 1235.904469] ? new_sync_read+0x740/0x740 [ 1235.909670] ? lock_acquire.part.0+0xbc/0x260 [ 1235.915357] ? ksys_write+0x116/0x250 [ 1235.920257] vfs_write+0x657/0x9b0 [ 1235.924871] ksys_write+0x116/0x250 [ 1235.929583] ? __ia32_sys_read+0xc0/0xc0 [ 1235.934783] ? kernfs_fop_llseek+0x160/0x1f0 [ 1235.940362] ? rcu_is_watching+0x15/0xd0 [ 1235.945563] do_syscall_64+0xf3/0x650 [ 1235.950469] ? irq_exit_rcu+0x1a/0x30 [ 1235.955368] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1235.961834] RIP: 0033:0x7f88d3a3722e [ 1235.966643] RSP: 002b:00007ffede7a1c70 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1235.975936] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f88d3a3722e [ 1235.984743] RDX: 0000000000000001 RSI: 000055e8626f86b0 RDI: 0000000000000007 [ 1235.993549] RBP: 00007ffede7a1c80 R08: 0000000000000000 R09: 0000000000000000 [ 1236.002357] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f88d38af390 [ 1236.011162] R13: 000055e8626f86b0 R14: 0000000000000007 R15: 0000000000000003 [ 1236.019972] [ 1236.023237] \x0aShowing all locks held in the system: [ 1236.031774] 1 lock held by khungtaskd/186: [ 1236.037166] #0: ffffffffa0efdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1236.048636] 3 locks held by systemd-journal/803: [ 1236.054613] #0: ff1100180e0b72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0 [ 1236.064498] #1: ffffffffa11029a0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120 [ 1236.077315] #2: ff11000189de1200 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0 [ 1236.088871] 4 locks held by python3/3710: [ 1236.094154] #0: ff1100012e7e13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.104041] #1: ff11000162a4be80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.115003] #2: ff11000120e02b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.126157] #3: ff11000120e69d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.136250] 4 locks held by python3/5740: [ 1236.141543] #0: ff1100012e7e13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1236.151432] #1: ff1100014fc12c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1236.162383] #2: ff11000120e02b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1236.173536] #3: ff11000120e69d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1236.183626] [ 1236.186092] =============================================\x0a [ 1358.351591] INFO: task python3:5740 blocked for more than 491 seconds. [ 1358.359748] Not tainted 7.1.0-rc4-ytun-g8252f1aece1a #1 [ 1358.366998] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1358.376591] task:python3 state:D stack:25672 pid:5740 tgid:5740 ppid:5739 task_flags:0x400100 flags:0x00080002 [ 1358.389798] Call Trace: [ 1358.393340] [ 1358.396490] __schedule+0xcc3/0x19c0 [ 1358.401307] ? io_schedule_timeout+0x140/0x140 [ 1358.407091] ? find_held_lock+0x2b/0x80 [ 1358.412197] ? __lock_release.isra.0+0x6b/0x1a0 [ 1358.418086] schedule+0xd8/0x260 [ 1358.422579] schedule_preempt_disabled+0x18/0x30 [ 1358.428558] __mutex_lock+0x6af/0x2010 [ 1358.433564] ? threaded_store+0x132/0x220 [ 1358.438933] ? ww_mutex_lock+0x160/0x160 [ 1358.444126] ? selinux_inode_free_security+0x330/0x330 [ 1358.450698] ? cap_capable+0x1d6/0x3e0 [ 1358.455774] ? threaded_store+0x132/0x220 [ 1358.461075] threaded_store+0x132/0x220 [ 1358.466180] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.471478] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.476780] ? sysfs_kf_write+0x65/0x160 [ 1358.481982] ? sysfs_file_ops+0x140/0x140 [ 1358.481986] kernfs_fop_write_iter+0x335/0x510 [ 1358.481990] ? kernfs_ops+0xf0/0xf0 [ 1358.481993] new_sync_write+0x333/0x750 [ 1358.481999] ? __lock_acquire+0x508/0xc10 [ 1358.508208] ? new_sync_read+0x740/0x740 [ 1358.513409] ? lock_acquire.part.0+0xbc/0x260 [ 1358.519095] ? ksys_write+0x116/0x250 [ 1358.524006] vfs_write+0x657/0x9b0 [ 1358.528622] ksys_write+0x116/0x250 [ 1358.533334] ? __ia32_sys_read+0xc0/0xc0 [ 1358.538536] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.544126] ? rcu_is_watching+0x15/0xd0 [ 1358.549330] do_syscall_64+0xf3/0x650 [ 1358.554236] ? trace_hardirqs_off+0xd/0x30 [ 1358.559634] ? exc_page_fault+0xda/0xf0 [ 1358.564735] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.571203] RIP: 0033:0x7f337193822e [ 1358.576012] RSP: 002b:00007fffcd73a940 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.585313] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f337193822e [ 1358.594123] RDX: 0000000000000001 RSI: 00005644224086b0 RDI: 0000000000000007 [ 1358.602929] RBP: 00007fffcd73a950 R08: 0000000000000000 R09: 0000000000000000 [ 1358.611737] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f33717b0390 [ 1358.620547] R13: 00005644224086b0 R14: 0000000000000007 R15: 0000000000000003 [ 1358.629359] [ 1358.632659] INFO: task python3:5740 is blocked on a mutex likely owned by task python3:3710. [ 1358.642933] task:python3 state:D stack:25672 pid:3710 tgid:3710 ppid:3709 task_flags:0x400100 flags:0x00080802 [ 1358.656134] Call Trace: [ 1358.659671] [ 1358.662823] __schedule+0xcc3/0x19c0 [ 1358.667627] ? io_schedule_timeout+0x140/0x140 [ 1358.673410] ? find_held_lock+0x2b/0x80 [ 1358.678515] ? __try_to_del_timer_sync+0x155/0x1f0 [ 1358.684696] schedule+0xd8/0x260 [ 1358.689116] schedule_timeout+0x13f/0x290 [ 1358.694415] ? hrtimer_nanosleep_restart+0x270/0x270 [ 1358.700784] ? trace_contention_end+0xb3/0x1a0 [ 1358.706569] ? hrtimers_cpu_dying+0x3e0/0x3e0 [ 1358.712258] ? ww_mutex_lock+0x160/0x160 [ 1358.717449] msleep+0x1b/0x30 [ 1358.721576] napi_set_threaded+0x3a5/0x440 [ 1358.726971] ? napi_resume_irqs+0x1a0/0x1a0 [ 1358.732465] netif_set_threaded+0x257/0x400 [ 1358.737957] threaded_store+0x1b8/0x220 [ 1358.743060] ? xps_rxqs_store+0x2e0/0x2e0 [ 1358.748350] ? sysfs_file_kobj+0xbd/0x1e0 [ 1358.753645] ? sysfs_kf_write+0x65/0x160 [ 1358.758843] ? sysfs_file_ops+0x140/0x140 [ 1358.764140] kernfs_fop_write_iter+0x335/0x510 [ 1358.769924] ? kernfs_ops+0xf0/0xf0 [ 1358.774635] new_sync_write+0x333/0x750 [ 1358.779740] ? __lock_acquire+0x508/0xc10 [ 1358.785038] ? new_sync_read+0x740/0x740 [ 1358.790240] ? lock_acquire.part.0+0xbc/0x260 [ 1358.795927] ? ksys_write+0x116/0x250 [ 1358.800838] vfs_write+0x657/0x9b0 [ 1358.805456] ksys_write+0x116/0x250 [ 1358.810167] ? __ia32_sys_read+0xc0/0xc0 [ 1358.815365] ? kernfs_fop_llseek+0x160/0x1f0 [ 1358.820955] ? rcu_is_watching+0x15/0xd0 [ 1358.826147] do_syscall_64+0xf3/0x650 [ 1358.831056] ? irq_exit_rcu+0x1a/0x30 [ 1358.835965] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 1358.842439] RIP: 0033:0x7f88d3a3722e [ 1358.847245] RSP: 002b:00007ffede7a1c70 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 1358.856543] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f88d3a3722e [ 1358.865351] RDX: 0000000000000001 RSI: 000055e8626f86b0 RDI: 0000000000000007 [ 1358.874157] RBP: 00007ffede7a1c80 R08: 0000000000000000 R09: 0000000000000000 [ 1358.882968] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f88d38af390 [ 1358.891777] R13: 000055e8626f86b0 R14: 0000000000000007 R15: 0000000000000003 [ 1358.900589] [ 1358.903858] \x0aShowing all locks held in the system: [ 1358.912398] 1 lock held by khungtaskd/186: [ 1358.917792] #0: ffffffffa0efdde0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30 [ 1358.929263] 3 locks held by systemd-journal/803: [ 1358.935242] #0: ff1100180dbb72e0 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x18d/0x19c0 [ 1358.945131] #1: ffffffffa11029a0 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7b/0x120 [ 1358.957950] #2: ff11000189de1780 (ptlock_ptr(ptdesc)#2){+.+.}-{3:3}, at: pte_offset_map_lock+0x114/0x2d0 [ 1358.969509] 4 locks held by python3/3710: [ 1358.974806] #0: ff1100012e7e13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1358.984689] #1: ff11000162a4be80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1358.995652] #2: ff11000120e02b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1359.006809] #3: ff11000120e69d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1359.016907] 4 locks held by python3/5740: [ 1359.022201] #0: ff1100012e7e13e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1359.032091] #1: ff1100014fc12c80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1359.043053] #2: ff11000120e02b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510 [ 1359.054203] #3: ff11000120e69d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220 [ 1359.064292] 5 locks held by (systemd-user-)/7598: [ 1359.070369] #0: ff1100012e8d13e8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x116/0x250 [ 1359.080261] #1: ff1100013d51c480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510 [ 1359.091222] #2: ffffffffa0f69b60 (cgroup_mutex){+.+.}-{4:4}, at: cgroup_kn_lock_live+0xe2/0x3e0 [ 1359.101891] #3: ffffffffa0d686f0 (cpu_hotplug_lock){++++}-{0:0}, at: cgroup_procs_write_start+0x3e5/0x650 [ 1359.113537] #4: ffffffffa0f698d0 (cgroup_threadgroup_rwsem){++++}-{0:0}, at: cgroup_procs_write_start+0x3f1/0x650 [ 1359.125963] [ 1359.128437] =============================================\x0a