[ 788.202490] nipa-hw-worker: drivers/net:napi_threaded.py: timed out
[ 788.204975] nipa-hw-worker: [5/47] Retrying drivers/net:napi_threaded.py
[ 788.335725] kselftest: Running tests in drivers/net
[ 989.663346] INFO: task python3:5758 blocked for more than 122 seconds.
[ 989.671547] Not tainted 7.1.0-rc3-oadl-g03c42cd70454 #1
[ 989.678799] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 989.688768] task:python3 state:D stack:25032 pid:5758 tgid:5758 ppid:5757 task_flags:0x400100 flags:0x00080000
[ 989.701976] Call Trace:
[ 989.705514]
[ 989.708665] __schedule+0xcc3/0x19c0
[ 989.713482] ? io_schedule_timeout+0x140/0x140
[ 989.719265] ? find_held_lock+0x2b/0x80
[ 989.724369] ? __lock_release.isra.0+0x6b/0x1a0
[ 989.730244] schedule+0xd4/0x260
[ 989.734659] schedule_preempt_disabled+0x18/0x30
[ 989.740637] __mutex_lock+0x6a3/0x2000
[ 989.745643] ? threaded_store+0x132/0x220
[ 989.750940] ? ww_mutex_lock+0x160/0x160
[ 989.756136] ? selinux_inode_free_security+0x330/0x330
[ 989.762704] ? cap_capable+0x1d6/0x3e0
[ 989.767710] ? threaded_store+0x132/0x220
[ 989.773005] threaded_store+0x132/0x220
[ 989.778105] ? xps_rxqs_store+0x2e0/0x2e0
[ 989.783404] ? sysfs_file_kobj+0xbd/0x1e0
[ 989.788704] ? sysfs_kf_write+0x65/0x160
[ 989.793903] ? sysfs_file_ops+0x140/0x140
[ 989.799198] kernfs_fop_write_iter+0x335/0x510
[ 989.804982] ? kernfs_ops+0xf0/0xf0
[ 989.809692] new_sync_write+0x333/0x750
[ 989.814796] ? __lock_acquire+0x508/0xc10
[ 989.820085] ? new_sync_read+0x740/0x740
[ 989.825285] ? lock_acquire.part.0+0xbc/0x260
[ 989.830970] ? ksys_write+0x116/0x250
[ 989.835881] vfs_write+0x657/0x9b0
[ 989.840497] ksys_write+0x116/0x250
[ 989.845208] ? __ia32_sys_read+0xc0/0xc0
[ 989.850404] ? kernfs_fop_llseek+0x160/0x1f0
[ 989.855992] ? rcu_is_watching+0x15/0xd0
[ 989.861193] do_syscall_64+0xf3/0x650
[ 989.866100] ? trace_hardirqs_off+0xd/0x30
[ 989.871485] ? exc_page_fault+0xda/0xf0
[ 989.876586] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 989.883061] RIP: 0033:0x7f352ac5722e
[ 989.887870] RSP: 002b:00007ffd3f08e130 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 989.897165] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f352ac5722e
[ 989.905971] RDX: 0000000000000001 RSI: 00005640e1ea3cd0 RDI: 0000000000000007
[ 989.914779] RBP: 00007ffd3f08e140 R08: 0000000000000000 R09: 0000000000000000
[ 989.923587] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f352aacf390
[ 989.932388] R13: 00005640e1ea3cd0 R14: 0000000000000007 R15: 0000000000000003
[ 989.941199]
[ 989.944481] INFO: task python3:5758 is blocked on a mutex likely owned by task python3:3728.
[ 989.954754] task:python3 state:D stack:25672 pid:3728 tgid:3728 ppid:3727 task_flags:0x400100 flags:0x00080802
[ 989.967956] Call Trace:
[ 989.971493]
[ 989.974644] __schedule+0xcc3/0x19c0
[ 989.979447] ? io_schedule_timeout+0x140/0x140
[ 989.985231] ? find_held_lock+0x2b/0x80
[ 989.990325] ? __try_to_del_timer_sync+0x155/0x1f0
[ 989.996493] schedule+0xd4/0x260
[ 990.000912] schedule_timeout+0x13f/0x290
[ 990.006207] ? hrtimer_nanosleep_restart+0x270/0x270
[ 990.012578] ? trace_contention_end+0xb3/0x1a0
[ 990.018360] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 990.024047] ? ww_mutex_lock+0x160/0x160
[ 990.029248] msleep+0x1b/0x30
[ 990.033373] napi_set_threaded+0x3a5/0x440
[ 990.038765] ? napi_resume_irqs+0x1a0/0x1a0
[ 990.044259] netif_set_threaded+0x257/0x400
[ 990.049749] threaded_store+0x1b8/0x220
[ 990.054849] ? xps_rxqs_store+0x2e0/0x2e0
[ 990.060144] ? sysfs_file_kobj+0xbd/0x1e0
[ 990.065438] ? sysfs_kf_write+0x65/0x160
[ 990.070635] ? sysfs_file_ops+0x140/0x140
[ 990.075927] kernfs_fop_write_iter+0x335/0x510
[ 990.081710] ? kernfs_ops+0xf0/0xf0
[ 990.086421] new_sync_write+0x333/0x750
[ 990.091521] ? __lock_acquire+0x508/0xc10
[ 990.096816] ? new_sync_read+0x740/0x740
[ 990.102016] ? lock_acquire.part.0+0xbc/0x260
[ 990.107701] ? ksys_write+0x116/0x250
[ 990.112610] vfs_write+0x657/0x9b0
[ 990.117221] ksys_write+0x116/0x250
[ 990.121929] ? __ia32_sys_read+0xc0/0xc0
[ 990.127126] ? kernfs_fop_llseek+0x160/0x1f0
[ 990.132714] ? rcu_is_watching+0x15/0xd0
[ 990.137914] do_syscall_64+0xf3/0x650
[ 990.142817] ? trace_hardirqs_off+0xd/0x30
[ 990.148211] ? exc_page_fault+0xda/0xf0
[ 990.153310] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 990.159775] RIP: 0033:0x7fb0d5dbd22e
[ 990.164584] RSP: 002b:00007ffd42109ec0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 990.173880] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb0d5dbd22e
[ 990.182680] RDX: 0000000000000001 RSI: 000055dd3c95ecd0 RDI: 0000000000000007
[ 990.191489] RBP: 00007ffd42109ed0 R08: 0000000000000000 R09: 0000000000000000
[ 990.200296] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fb0d5c35390
[ 990.209104] R13: 000055dd3c95ecd0 R14: 0000000000000007 R15: 0000000000000003
[ 990.217907]
[ 990.221165] \x0aShowing all locks held in the system:
[ 990.229700] 1 lock held by khungtaskd/186:
[ 990.235093] #0: ffffffffb4efe060 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 990.246575] 4 locks held by python3/3728:
[ 990.251869] #0: ff1100012ecdc3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.261759] #1: ff1100016b888e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.272719] #2: ff110001221b6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.283875] #3: ff11000122239d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.293966] 4 locks held by python3/5758:
[ 990.299260] #0: ff1100012ecdc3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 990.309148] #1: ff11000231684a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 990.320107] #2: ff110001221b6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 990.331266] #3: ff11000122239d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 990.341355]
[ 990.343820] =============================================\x0a
[ 1112.543545] INFO: task python3:5758 blocked for more than 245 seconds.
[ 1112.551694] Not tainted 7.1.0-rc3-oadl-g03c42cd70454 #1
[ 1112.558943] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1112.568534] task:python3 state:D stack:25032 pid:5758 tgid:5758 ppid:5757 task_flags:0x400100 flags:0x00080000
[ 1112.581739] Call Trace:
[ 1112.585279]
[ 1112.588432] __schedule+0xcc3/0x19c0
[ 1112.593249] ? io_schedule_timeout+0x140/0x140
[ 1112.599037] ? find_held_lock+0x2b/0x80
[ 1112.604145] ? __lock_release.isra.0+0x6b/0x1a0
[ 1112.610028] schedule+0xd4/0x260
[ 1112.614445] schedule_preempt_disabled+0x18/0x30
[ 1112.620427] __mutex_lock+0x6a3/0x2000
[ 1112.625434] ? threaded_store+0x132/0x220
[ 1112.630736] ? ww_mutex_lock+0x160/0x160
[ 1112.635934] ? selinux_inode_free_security+0x330/0x330
[ 1112.642503] ? cap_capable+0x1d6/0x3e0
[ 1112.647512] ? threaded_store+0x132/0x220
[ 1112.652808] threaded_store+0x132/0x220
[ 1112.657913] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.663213] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.668512] ? sysfs_kf_write+0x65/0x160
[ 1112.673703] ? sysfs_file_ops+0x140/0x140
[ 1112.678999] kernfs_fop_write_iter+0x335/0x510
[ 1112.684787] ? kernfs_ops+0xf0/0xf0
[ 1112.689498] new_sync_write+0x333/0x750
[ 1112.694602] ? __lock_acquire+0x508/0xc10
[ 1112.699897] ? new_sync_read+0x740/0x740
[ 1112.705099] ? lock_acquire.part.0+0xbc/0x260
[ 1112.710784] ? ksys_write+0x116/0x250
[ 1112.715693] vfs_write+0x657/0x9b0
[ 1112.720310] ksys_write+0x116/0x250
[ 1112.725023] ? __ia32_sys_read+0xc0/0xc0
[ 1112.730224] ? kernfs_fop_llseek+0x160/0x1f0
[ 1112.735814] ? rcu_is_watching+0x15/0xd0
[ 1112.741014] do_syscall_64+0xf3/0x650
[ 1112.745921] ? trace_hardirqs_off+0xd/0x30
[ 1112.751316] ? exc_page_fault+0xda/0xf0
[ 1112.756419] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1112.762877] RIP: 0033:0x7f352ac5722e
[ 1112.767687] RSP: 002b:00007ffd3f08e130 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1112.776979] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f352ac5722e
[ 1112.785787] RDX: 0000000000000001 RSI: 00005640e1ea3cd0 RDI: 0000000000000007
[ 1112.794587] RBP: 00007ffd3f08e140 R08: 0000000000000000 R09: 0000000000000000
[ 1112.803397] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f352aacf390
[ 1112.812205] R13: 00005640e1ea3cd0 R14: 0000000000000007 R15: 0000000000000003
[ 1112.821020]
[ 1112.824305] INFO: task python3:5758 is blocked on a mutex likely owned by task python3:3728.
[ 1112.834579] task:python3 state:D stack:25672 pid:3728 tgid:3728 ppid:3727 task_flags:0x400100 flags:0x00080802
[ 1112.847784] Call Trace:
[ 1112.851322]
[ 1112.854474] __schedule+0xcc3/0x19c0
[ 1112.859285] ? io_schedule_timeout+0x140/0x140
[ 1112.865070] ? find_held_lock+0x2b/0x80
[ 1112.870166] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1112.876343] schedule+0xd4/0x260
[ 1112.880762] schedule_timeout+0x13f/0x290
[ 1112.886053] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1112.892423] ? trace_contention_end+0xb3/0x1a0
[ 1112.898199] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1112.903890] ? ww_mutex_lock+0x160/0x160
[ 1112.909093] msleep+0x1b/0x30
[ 1112.913218] napi_set_threaded+0x3a5/0x440
[ 1112.918616] ? napi_resume_irqs+0x1a0/0x1a0
[ 1112.924110] netif_set_threaded+0x257/0x400
[ 1112.929604] threaded_store+0x1b8/0x220
[ 1112.934708] ? xps_rxqs_store+0x2e0/0x2e0
[ 1112.940007] ? sysfs_file_kobj+0xbd/0x1e0
[ 1112.945306] ? sysfs_kf_write+0x65/0x160
[ 1112.950504] ? sysfs_file_ops+0x140/0x140
[ 1112.955800] kernfs_fop_write_iter+0x335/0x510
[ 1112.961583] ? kernfs_ops+0xf0/0xf0
[ 1112.966294] new_sync_write+0x333/0x750
[ 1112.971398] ? __lock_acquire+0x508/0xc10
[ 1112.976693] ? new_sync_read+0x740/0x740
[ 1112.981893] ? lock_acquire.part.0+0xbc/0x260
[ 1112.987577] ? ksys_write+0x116/0x250
[ 1112.992487] vfs_write+0x657/0x9b0
[ 1112.997102] ksys_write+0x116/0x250
[ 1113.001816] ? __ia32_sys_read+0xc0/0xc0
[ 1113.007017] ? kernfs_fop_llseek+0x160/0x1f0
[ 1113.012597] ? rcu_is_watching+0x15/0xd0
[ 1113.017792] do_syscall_64+0xf3/0x650
[ 1113.022697] ? trace_hardirqs_off+0xd/0x30
[ 1113.028091] ? exc_page_fault+0xda/0xf0
[ 1113.033194] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1113.039662] RIP: 0033:0x7fb0d5dbd22e
[ 1113.044472] RSP: 002b:00007ffd42109ec0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1113.053764] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb0d5dbd22e
[ 1113.062572] RDX: 0000000000000001 RSI: 000055dd3c95ecd0 RDI: 0000000000000007
[ 1113.071372] RBP: 00007ffd42109ed0 R08: 0000000000000000 R09: 0000000000000000
[ 1113.080181] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fb0d5c35390
[ 1113.088991] R13: 000055dd3c95ecd0 R14: 0000000000000007 R15: 0000000000000003
[ 1113.097804]
[ 1113.101066] \x0aShowing all locks held in the system:
[ 1113.109601] 1 lock held by khungtaskd/186:
[ 1113.114997] #0: ffffffffb4efe060 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1113.126468] 4 locks held by python3/3728:
[ 1113.131766] #0: ff1100012ecdc3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.141650] #1: ff1100016b888e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.152611] #2: ff110001221b6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.163767] #3: ff11000122239d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.173864] 4 locks held by python3/5758:
[ 1113.179160] #0: ff1100012ecdc3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1113.189051] #1: ff11000231684a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1113.200012] #2: ff110001221b6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1113.211164] #3: ff11000122239d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1113.221258]
[ 1113.223726] =============================================\x0a
[ 1235.424758] INFO: task python3:5758 blocked for more than 368 seconds.
[ 1235.432908] Not tainted 7.1.0-rc3-oadl-g03c42cd70454 #1
[ 1235.440158] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1235.449749] task:python3 state:D stack:25032 pid:5758 tgid:5758 ppid:5757 task_flags:0x400100 flags:0x00080002
[ 1235.462955] Call Trace:
[ 1235.466495]
[ 1235.469648] __schedule+0xcc3/0x19c0
[ 1235.474466] ? io_schedule_timeout+0x140/0x140
[ 1235.480253] ? find_held_lock+0x2b/0x80
[ 1235.485359] ? __lock_release.isra.0+0x6b/0x1a0
[ 1235.491243] schedule+0xd4/0x260
[ 1235.495661] schedule_preempt_disabled+0x18/0x30
[ 1235.501642] __mutex_lock+0x6a3/0x2000
[ 1235.506645] ? threaded_store+0x132/0x220
[ 1235.511944] ? ww_mutex_lock+0x160/0x160
[ 1235.517142] ? selinux_inode_free_security+0x330/0x330
[ 1235.523715] ? cap_capable+0x1d6/0x3e0
[ 1235.528727] ? threaded_store+0x132/0x220
[ 1235.534024] threaded_store+0x132/0x220
[ 1235.539119] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.544416] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.549717] ? sysfs_kf_write+0x65/0x160
[ 1235.554917] ? sysfs_file_ops+0x140/0x140
[ 1235.560213] kernfs_fop_write_iter+0x335/0x510
[ 1235.566001] ? kernfs_ops+0xf0/0xf0
[ 1235.570711] new_sync_write+0x333/0x750
[ 1235.575815] ? __lock_acquire+0x508/0xc10
[ 1235.581113] ? new_sync_read+0x740/0x740
[ 1235.586315] ? lock_acquire.part.0+0xbc/0x260
[ 1235.592003] ? ksys_write+0x116/0x250
[ 1235.596915] vfs_write+0x657/0x9b0
[ 1235.601532] ksys_write+0x116/0x250
[ 1235.606244] ? __ia32_sys_read+0xc0/0xc0
[ 1235.611446] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.617034] ? rcu_is_watching+0x15/0xd0
[ 1235.622229] do_syscall_64+0xf3/0x650
[ 1235.627137] ? trace_hardirqs_off+0xd/0x30
[ 1235.632533] ? exc_page_fault+0xda/0xf0
[ 1235.637635] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.644095] RIP: 0033:0x7f352ac5722e
[ 1235.648905] RSP: 002b:00007ffd3f08e130 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.658194] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f352ac5722e
[ 1235.666993] RDX: 0000000000000001 RSI: 00005640e1ea3cd0 RDI: 0000000000000007
[ 1235.675801] RBP: 00007ffd3f08e140 R08: 0000000000000000 R09: 0000000000000000
[ 1235.684612] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f352aacf390
[ 1235.693420] R13: 00005640e1ea3cd0 R14: 0000000000000007 R15: 0000000000000003
[ 1235.702233]
[ 1235.705530] INFO: task python3:5758 is blocked on a mutex likely owned by task python3:3728.
[ 1235.715804] task:python3 state:D stack:25672 pid:3728 tgid:3728 ppid:3727 task_flags:0x400100 flags:0x00080802
[ 1235.729007] Call Trace:
[ 1235.732548]
[ 1235.735701] __schedule+0xcc3/0x19c0
[ 1235.740512] ? io_schedule_timeout+0x140/0x140
[ 1235.746298] ? find_held_lock+0x2b/0x80
[ 1235.751400] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1235.757581] schedule+0xd4/0x260
[ 1235.762000] schedule_timeout+0x13f/0x290
[ 1235.767288] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1235.773657] ? trace_contention_end+0xb3/0x1a0
[ 1235.779444] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1235.785135] ? ww_mutex_lock+0x160/0x160
[ 1235.790336] msleep+0x1b/0x30
[ 1235.794460] napi_set_threaded+0x3a5/0x440
[ 1235.799858] ? napi_resume_irqs+0x1a0/0x1a0
[ 1235.805354] netif_set_threaded+0x257/0x400
[ 1235.810846] threaded_store+0x1b8/0x220
[ 1235.815951] ? xps_rxqs_store+0x2e0/0x2e0
[ 1235.821249] ? sysfs_file_kobj+0xbd/0x1e0
[ 1235.826537] ? sysfs_kf_write+0x65/0x160
[ 1235.831738] ? sysfs_file_ops+0x140/0x140
[ 1235.837033] kernfs_fop_write_iter+0x335/0x510
[ 1235.842819] ? kernfs_ops+0xf0/0xf0
[ 1235.847529] new_sync_write+0x333/0x750
[ 1235.852632] ? __lock_acquire+0x508/0xc10
[ 1235.857927] ? new_sync_read+0x740/0x740
[ 1235.863127] ? lock_acquire.part.0+0xbc/0x260
[ 1235.868811] ? ksys_write+0x116/0x250
[ 1235.873723] vfs_write+0x657/0x9b0
[ 1235.878338] ksys_write+0x116/0x250
[ 1235.883042] ? __ia32_sys_read+0xc0/0xc0
[ 1235.888234] ? kernfs_fop_llseek+0x160/0x1f0
[ 1235.893825] ? rcu_is_watching+0x15/0xd0
[ 1235.899023] do_syscall_64+0xf3/0x650
[ 1235.903933] ? trace_hardirqs_off+0xd/0x30
[ 1235.909326] ? exc_page_fault+0xda/0xf0
[ 1235.914429] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1235.920888] RIP: 0033:0x7fb0d5dbd22e
[ 1235.925689] RSP: 002b:00007ffd42109ec0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1235.934985] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb0d5dbd22e
[ 1235.943787] RDX: 0000000000000001 RSI: 000055dd3c95ecd0 RDI: 0000000000000007
[ 1235.952601] RBP: 00007ffd42109ed0 R08: 0000000000000000 R09: 0000000000000000
[ 1235.961412] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fb0d5c35390
[ 1235.970221] R13: 000055dd3c95ecd0 R14: 0000000000000007 R15: 0000000000000003
[ 1235.979034]
[ 1235.982299] \x0aShowing all locks held in the system:
[ 1235.990836] 1 lock held by khungtaskd/186:
[ 1235.996232] #0: ffffffffb4efe060 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1236.007706] 4 locks held by python3/3728:
[ 1236.013003] #0: ff1100012ecdc3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.022893] #1: ff1100016b888e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.033845] #2: ff110001221b6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.045001] #3: ff11000122239d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.055096] 4 locks held by python3/5758:
[ 1236.060392] #0: ff1100012ecdc3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1236.070282] #1: ff11000231684a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1236.081244] #2: ff110001221b6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1236.092399] #3: ff11000122239d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1236.102493]
[ 1236.104960] =============================================\x0a
[ 1358.306962] INFO: task python3:5758 blocked for more than 491 seconds.
[ 1358.315110] Not tainted 7.1.0-rc3-oadl-g03c42cd70454 #1
[ 1358.322360] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1358.331952] task:python3 state:D stack:25032 pid:5758 tgid:5758 ppid:5757 task_flags:0x400100 flags:0x00080002
[ 1358.345160] Call Trace:
[ 1358.348700]
[ 1358.351853] __schedule+0xcc3/0x19c0
[ 1358.356673] ? io_schedule_timeout+0x140/0x140
[ 1358.362460] ? find_held_lock+0x2b/0x80
[ 1358.367569] ? __lock_release.isra.0+0x6b/0x1a0
[ 1358.373452] schedule+0xd4/0x260
[ 1358.377860] schedule_preempt_disabled+0x18/0x30
[ 1358.383840] __mutex_lock+0x6a3/0x2000
[ 1358.388838] ? threaded_store+0x132/0x220
[ 1358.394140] ? ww_mutex_lock+0x160/0x160
[ 1358.399339] ? selinux_inode_free_security+0x330/0x330
[ 1358.405908] ? cap_capable+0x1d6/0x3e0
[ 1358.410916] ? threaded_store+0x132/0x220
[ 1358.416214] threaded_store+0x132/0x220
[ 1358.421307] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.426608] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.431909] ? sysfs_kf_write+0x65/0x160
[ 1358.437109] ? sysfs_file_ops+0x140/0x140
[ 1358.442397] kernfs_fop_write_iter+0x335/0x510
[ 1358.448183] ? kernfs_ops+0xf0/0xf0
[ 1358.452894] new_sync_write+0x333/0x750
[ 1358.457989] ? __lock_acquire+0x508/0xc10
[ 1358.463285] ? new_sync_read+0x740/0x740
[ 1358.468488] ? lock_acquire.part.0+0xbc/0x260
[ 1358.474174] ? ksys_write+0x116/0x250
[ 1358.479075] vfs_write+0x657/0x9b0
[ 1358.483692] ksys_write+0x116/0x250
[ 1358.488406] ? __ia32_sys_read+0xc0/0xc0
[ 1358.493604] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.499195] ? rcu_is_watching+0x15/0xd0
[ 1358.504399] do_syscall_64+0xf3/0x650
[ 1358.509304] ? trace_hardirqs_off+0xd/0x30
[ 1358.514699] ? exc_page_fault+0xda/0xf0
[ 1358.519795] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.526266] RIP: 0033:0x7f352ac5722e
[ 1358.531077] RSP: 002b:00007ffd3f08e130 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.540374] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f352ac5722e
[ 1358.549173] RDX: 0000000000000001 RSI: 00005640e1ea3cd0 RDI: 0000000000000007
[ 1358.557973] RBP: 00007ffd3f08e140 R08: 0000000000000000 R09: 0000000000000000
[ 1358.566783] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f352aacf390
[ 1358.575593] R13: 00005640e1ea3cd0 R14: 0000000000000007 R15: 0000000000000003
[ 1358.584407]
[ 1358.587683] INFO: task python3:5758 is blocked on a mutex likely owned by task python3:3728.
[ 1358.597955] task:python3 state:D stack:25672 pid:3728 tgid:3728 ppid:3727 task_flags:0x400100 flags:0x00080802
[ 1358.611156] Call Trace:
[ 1358.614695]
[ 1358.617847] __schedule+0xcc3/0x19c0
[ 1358.622660] ? io_schedule_timeout+0x140/0x140
[ 1358.628445] ? find_held_lock+0x2b/0x80
[ 1358.633552] ? __try_to_del_timer_sync+0x155/0x1f0
[ 1358.639729] schedule+0xd4/0x260
[ 1358.644148] schedule_timeout+0x13f/0x290
[ 1358.649437] ? hrtimer_nanosleep_restart+0x270/0x270
[ 1358.655811] ? trace_contention_end+0xb3/0x1a0
[ 1358.661599] ? hrtimers_cpu_dying+0x3e0/0x3e0
[ 1358.667288] ? ww_mutex_lock+0x160/0x160
[ 1358.672488] msleep+0x1b/0x30
[ 1358.676615] napi_set_threaded+0x3a5/0x440
[ 1358.682011] ? napi_resume_irqs+0x1a0/0x1a0
[ 1358.687504] netif_set_threaded+0x257/0x400
[ 1358.693000] threaded_store+0x1b8/0x220
[ 1358.698105] ? xps_rxqs_store+0x2e0/0x2e0
[ 1358.703402] ? sysfs_file_kobj+0xbd/0x1e0
[ 1358.708700] ? sysfs_kf_write+0x65/0x160
[ 1358.713892] ? sysfs_file_ops+0x140/0x140
[ 1358.719180] kernfs_fop_write_iter+0x335/0x510
[ 1358.724963] ? kernfs_ops+0xf0/0xf0
[ 1358.729673] new_sync_write+0x333/0x750
[ 1358.734778] ? __lock_acquire+0x508/0xc10
[ 1358.740075] ? new_sync_read+0x740/0x740
[ 1358.745276] ? lock_acquire.part.0+0xbc/0x260
[ 1358.750963] ? ksys_write+0x116/0x250
[ 1358.755872] vfs_write+0x657/0x9b0
[ 1358.760490] ksys_write+0x116/0x250
[ 1358.765205] ? __ia32_sys_read+0xc0/0xc0
[ 1358.770402] ? kernfs_fop_llseek+0x160/0x1f0
[ 1358.775994] ? rcu_is_watching+0x15/0xd0
[ 1358.781193] do_syscall_64+0xf3/0x650
[ 1358.786098] ? trace_hardirqs_off+0xd/0x30
[ 1358.791493] ? exc_page_fault+0xda/0xf0
[ 1358.796586] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[ 1358.803051] RIP: 0033:0x7fb0d5dbd22e
[ 1358.807861] RSP: 002b:00007ffd42109ec0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1358.817160] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb0d5dbd22e
[ 1358.825970] RDX: 0000000000000001 RSI: 000055dd3c95ecd0 RDI: 0000000000000007
[ 1358.834779] RBP: 00007ffd42109ed0 R08: 0000000000000000 R09: 0000000000000000
[ 1358.843590] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fb0d5c35390
[ 1358.852398] R13: 000055dd3c95ecd0 R14: 0000000000000007 R15: 0000000000000003
[ 1358.861213]
[ 1358.864478] \x0aShowing all locks held in the system:
[ 1358.873016] 1 lock held by khungtaskd/186:
[ 1358.878411] #0: ffffffffb4efe060 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[ 1358.889891] 4 locks held by python3/3728:
[ 1358.895188] #0: ff1100012ecdc3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.905081] #1: ff1100016b888e80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.916045] #2: ff110001221b6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.927202] #3: ff11000122239d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.937297] 4 locks held by python3/5758:
[ 1358.942595] #0: ff1100012ecdc3e8 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x116/0x250
[ 1358.952485] #1: ff11000231684a80 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x248/0x510
[ 1358.963436] #2: ff110001221b6b50 (kn->active#118){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x29e/0x510
[ 1358.974597] #3: ff11000122239d68 (&dev->lock){+.+.}-{4:4}, at: threaded_store+0x132/0x220
[ 1358.984690]
[ 1358.987158] =============================================\x0a