virtme: waiting for virtiofsd to start virtme: use 'microvm' QEMU architecture [ 1.111721][ T1] tun: Universal TUN/TAP device driver, 1.6 [ 1.111741][ T1] BUG: sleeping function called from invalid context at ./include/linux/sched/mm.h:322 [ 1.111743][ T1] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 1, name: swapper/0 [ 1.111744][ T1] preempt_count: 1, expected: 0 [ 1.111745][ T1] RCU nest depth: 0, expected: 0 [ 1.111746][ T1] locks held by swapper/0/1: 4, last CPU#3: [ 1.111748][ T1] #0: ffffffffa497ddc0 (console_lock){+.+.}-{0:0}, at: vprintk_emit+0x320/0x3e0 [ 1.111759][ T1] #1: ffffffffa497de38 (console_srcu){....}-{0:0}, at: console_flush_one_record+0x106/0x710 [ 1.111763][ T1] #2: ffffffffa489d760 (console_owner){....}-{0:0}, at: console_lock_spinning_enable+0x40/0x60 [ 1.111767][ T1] #3: ffffffffa489d660 (printk_legacy_map-wait-type-override){....}-{3:3}, at: console_emit_next_record+0x1df/0x4c0 [ 1.111771][ T1] irq event stamp: 317308 [ 1.111772][ T1] hardirqs last enabled at (317307): [] __down_trylock_console_sem+0x86/0xa0 [ 1.111775][ T1] hardirqs last disabled at (317308): [] console_emit_next_record+0x3d4/0x4c0 [ 1.111777][ T1] softirqs last enabled at (317196): [] __alloc_skb+0x4c2/0x5f0 [ 1.111781][ T1] softirqs last disabled at (317194): [] __alloc_skb+0x4c2/0x5f0 [ 1.111783][ T1] Preemption disabled at: [ 1.111784][ T1] [] vprintk_emit+0x31b/0x3e0 [ 1.111788][ T1] CPU: 3 UID: 0 PID: 1 Comm: swapper/0 Not tainted 7.2.0-virtme #1 PREEMPT(full) [ 1.111791][ T1] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 1.111792][ T1] Call Trace: [ 1.111794][ T1] [ 1.111797][ T1] dump_stack_lvl+0x6f/0xa0 [ 1.111803][ T1] ? vprintk_emit+0x31b/0x3e0 [ 1.111804][ T1] __might_resched.cold+0x1fe/0x2c1 [ 1.111808][ T1] ? perf_trace_sched_switch+0x7c0/0x7c0 [ 1.111812][ T1] ? __kmalloc_noprof+0xdb/0x760 [ 1.111817][ T1] __kmalloc_noprof+0x443/0x760 [ 1.111819][ T1] ? alloc_buf.isra.0+0x4b/0x260 [ 1.111824][ T1] ? do_raw_spin_unlock+0x59/0x250 [ 1.111827][ T1] alloc_buf.isra.0+0x4b/0x260 [ 1.111830][ T1] put_chars+0x1e1/0x2f0 [ 1.111832][ T1] ? __send_to_port+0x420/0x420 [ 1.111835][ T1] ? rcu_read_lock_any_held+0x3c/0x90 [ 1.111838][ T1] ? validate_chain+0x38b/0xc20 [ 1.111842][ T1] hvc_console_print+0x292/0x780 [ 1.111846][ T1] ? __lock_acquire+0x518/0xc20 [ 1.111848][ T1] ? __lock_acquire+0x518/0xc20 [ 1.111852][ T1] ? hvc_write+0x3a0/0x3a0 [ 1.111855][ T1] ? rcu_is_watching+0x16/0xd0 [ 1.111858][ T1] ? lock_acquire+0x13c/0x160 [ 1.111862][ T1] console_emit_next_record+0x22f/0x4c0 [ 1.111865][ T1] ? devkmsg_read+0x4b0/0x4b0 [ 1.111867][ T1] ? console_flush_one_record+0x106/0x710 [ 1.111870][ T1] ? rcu_is_watching+0x16/0xd0 [ 1.111872][ T1] ? lock_acquire+0x13c/0x160 [ 1.111875][ T1] console_flush_one_record+0x46f/0x710 [ 1.111879][ T1] ? console_emit_next_record+0x4c0/0x4c0 [ 1.111881][ T1] ? __lock_acquire+0x518/0xc20 [ 1.111886][ T1] console_unlock+0xee/0x1f0 [ 1.111889][ T1] ? console_flush_one_record+0x710/0x710 [ 1.111891][ T1] ? rcu_is_watching+0x16/0xd0 [ 1.111892][ T1] ? lock_acquire+0xe0/0x160 [ 1.111896][ T1] ? __down_trylock_console_sem+0x5e/0xa0 [ 1.111898][ T1] ? vprintk_emit+0x320/0x3e0 [ 1.111900][ T1] vprintk_emit+0x37c/0x3e0 [ 1.111903][ T1] ? wake_up_klogd_work_func+0x90/0x90 [ 1.111907][ T1] ? team_module_init+0x60/0x60 [ 1.111911][ T1] _printk+0xc7/0x100 [ 1.111914][ T1] ? snapshot_read.cold+0x21/0x21 [ 1.111916][ T1] ? lock_pin_lock+0x80/0x180 [ 1.111919][ T1] ? _raw_spin_unlock_irqrestore+0x40/0x80 [ 1.111922][ T1] ? add_device_randomness+0xbb/0x100 [ 1.111925][ T1] ? random_write_iter+0x20/0x20 [ 1.111927][ T1] ? phy_module_init+0x20/0x20 [ 1.111930][ T1] ? do_one_initcall+0x113/0x4f0 [ 1.111933][ T1] tun_init+0x24/0xb0 [ 1.111935][ T1] do_one_initcall+0x124/0x4f0 [ 1.111937][ T1] ? trace_event_raw_event_initcall_level+0x280/0x280 [ 1.111939][ T1] ? parameq+0x110/0x110 [ 1.111944][ T1] ? kernel_init_freeable+0x3f1/0x630 [ 1.111948][ T1] ? rcu_is_watching+0x16/0xd0 [ 1.111951][ T1] kernel_init_freeable+0x596/0x630 [ 1.111954][ T1] ? rest_init+0x280/0x280 [ 1.111957][ T1] kernel_init+0x21/0x150 [ 1.111959][ T1] ? rest_init+0x280/0x280 [ 1.111961][ T1] ret_from_fork+0x474/0x6b0 [ 1.111965][ T1] ? arch_exit_to_user_mode_prepare.isra.0+0x120/0x120 [ 1.111968][ T1] ? __switch_to+0x5a3/0xe00 [ 1.111972][ T1] ? rest_init+0x280/0x280 [ 1.111974][ T1] ret_from_fork_asm+0x11/0x20 [ 1.111981][ T1] [ 1.130167][ T1] i8042: PNP: No PS/2 controller found. [ 1.138122][ T1] rtc_cmos PNP0B00:00: registered as rtc0 [ 1.138692][ T1] rtc_cmos PNP0B00:00: setting system clock to 2026-08-28T03:58:18 UTC (1787889498) [ 1.139592][ T1] rtc_cmos PNP0B00:00: alarms up to one day, 242 bytes nvram [ 1.142427][ T1] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev [ 1.148630][ T1] ipip: IPv4 and MPLS over IPv4 tunneling driver [ 1.152202][ T1] IPv4 over IPsec tunneling driver [ 1.155145][ T1] NET: Registered PF_INET6 protocol family [ 1.160631][ T1] Segment Routing with IPv6 [ 1.160809][ T1] RPL Segment Routing with IPv6 [ 1.161249][ T1] In-situ OAM (IOAM) with IPv6 [ 1.164630][ T1] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver [ 1.171333][ T1] NET: Registered PF_PACKET protocol family [ 1.171730][ T1] 8021q: 802.1Q VLAN Support v1.8 [ 1.172131][ T1] sctp: Hash tables configured (bind 32/56) [ 1.172752][ T1] 9pnet: Installing 9P2000 support [ 1.173100][ T1] Key type dns_resolver registered [ 1.173748][ T1] NET: Registered PF_VSOCK protocol family [ 1.178349][ T1] IPI shorthand broadcast: enabled [ 1.287808][ T1] sched_clock: Marking stable (1252001592, 35237478)->(1386486327, -99247257) [ 1.290470][ T1] registered taskstats version 1 [ 1.293783][ T1] Loading compiled-in X.509 certificates [ 1.404311][ T1] Demotion targets for Node 0: null [ 1.404609][ T1] kmemleak: Kernel memory leak detector initialized (mem pool available: 11805) [ 1.404909][ T1] page_owner is disabled [ 1.406057][ T1] PM: Magic number: 6:874:968 [ 1.406536][ T1] netconsole: network logging started [ 1.408786][ T1] ALSA device list: [ 1.409454][ T1] No soundcards found. [ 1.410821][ T1] md: Skipping autodetection of RAID arrays. (raid=autodetect will force) [ 1.413700][ T1] VFS: Mounted root (virtiofs filesystem) readonly on device 0:25. [ 1.414578][ T1] devtmpfs: mounted [ 1.414866][ T1] VFS: Pivoted into new rootfs [ 1.444841][ T1] Freeing unused kernel image (initmem) memory: 2624K [ 1.445066][ T1] Write protecting the kernel read-only data: 65536k [ 1.445461][ T1] Freeing unused kernel image (text/rodata gap) memory: 100K [ 1.445794][ T1] Freeing unused kernel image (rodata/data gap) memory: 312K [ 1.446150][ T1] Run /usr/lib/python3.14/site-packages/virtme/guest/bin/virtme-ng-init as init process [ 1.446432][ T1] with arguments: [ 1.446551][ T1] /usr/lib/python3.14/site-packages/virtme/guest/bin/virtme-ng-init [ 1.446780][ T1] with environment: [ 1.446904][ T1] HOME=/ [ 1.447025][ T1] TERM=dumb [ 1.447148][ T1] virtme_hostname=vmksft-net-dbg,debug-threads=on [ 1.447348][ T1] nr_open=2147483584 [ 1.447467][ T1] virtme_link_mods=/srv/vmksft/testing/wt-2/.virtme_mods/lib/modules/0.0.0 [ 1.447829][ T1] virtme_rw_overlay0=/etc [ 1.448039][ T1] virtme_rw_overlay1=/lib [ 1.448210][ T1] virtme_rw_overlay2=/home [ 1.450249][ T1] virtme_rw_overlay3=/opt [ 1.450421][ T1] virtme_rw_overlay4=/srv [ 1.450576][ T1] virtme_rw_overlay5=/usr [ 1.450726][ T1] virtme_rw_overlay6=/var [ 1.450899][ T1] virtme_rw_overlay7=/tmp [ 1.451057][ T1] virtme_console=ttyS0 [ 1.451209][ T1] virtme_chdir=srv/vmksft/testing/wt-2 [ 1.466460][ T1] virtme-ng-init: mount devtmpfs -> /dev: EBUSY: Device or resource busy [ 1.467820][ T1] virtme-ng-init: Setting hostname to vmksft-net-dbg,debug-threads=on... [ 1.478966][ T1] overlayfs: failed to set xattr on upper [ 1.479268][ T1] overlayfs: ...falling back to redirect_dir=nofollow. [ 1.479539][ T1] overlayfs: ...falling back to uuid=null. [ 1.481909][ T1] overlayfs: failed to set xattr on upper [ 1.482116][ T1] overlayfs: ...falling back to redirect_dir=nofollow. [ 1.482357][ T1] overlayfs: ...falling back to uuid=null. [ 1.484317][ T1] overlayfs: failed to set xattr on upper [ 1.484512][ T1] overlayfs: ...falling back to redirect_dir=nofollow. [ 1.484750][ T1] overlayfs: ...falling back to uuid=null. [ 1.487100][ T1] overlayfs: failed to set xattr on upper [ 1.487304][ T1] overlayfs: ...falling back to redirect_dir=nofollow. [ 1.487536][ T1] overlayfs: ...falling back to uuid=null. [ 1.489454][ T1] overlayfs: failed to set xattr on upper [ 1.489644][ T1] overlayfs: ...falling back to redirect_dir=nofollow. [ 1.489877][ T1] overlayfs: ...falling back to uuid=null. [ 1.491653][ T1] overlayfs: failed to set xattr on upper [ 1.491850][ T1] overlayfs: ...falling back to redirect_dir=nofollow. [ 1.492092][ T1] overlayfs: ...falling back to uuid=null. [ 1.494153][ T1] overlayfs: failed to set xattr on upper [ 1.494353][ T1] overlayfs: ...falling back to redirect_dir=nofollow. [ 1.494592][ T1] overlayfs: ...falling back to uuid=null. [ 1.496553][ T1] overlayfs: failed to set xattr on upper [ 1.496746][ T1] overlayfs: ...falling back to redirect_dir=nofollow. [ 1.496990][ T1] overlayfs: ...falling back to uuid=null. [ 1.509179][ T1] virtme-ng-init: running systemd-tmpfiles [ 1.529007][ T75] kwatchdog (75) used greatest stack depth: 29688 bytes left [ 3.557477][ T74] systemd-tmpfile (74) used greatest stack depth: 24896 bytes left [ 3.557493][ T74] BUG: sleeping function called from invalid context at ./include/linux/sched/mm.h:322 [ 3.557495][ T74] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 74, name: systemd-tmpfile [ 3.557497][ T74] preempt_count: 2, expected: 0 [ 3.557497][ T74] RCU nest depth: 0, expected: 0 [ 3.557498][ T74] locks held by systemd-tmpfile/74: 5, last CPU#1: [ 3.557500][ T74] #0: ffffffffa48167b8 (low_water_lock){+.+.}-{3:3}, at: do_exit+0x887/0xdc0 [ 3.557512][ T74] #1: ffffffffa497ddc0 (console_lock){+.+.}-{0:0}, at: vprintk_emit+0x320/0x3e0 [ 3.557516][ T74] #2: ffffffffa497de38 (console_srcu){....}-{0:0}, at: console_flush_one_record+0x106/0x710 [ 3.557520][ T74] #3: ffffffffa489d760 (console_owner){....}-{0:0}, at: console_lock_spinning_enable+0x40/0x60 [ 3.557524][ T74] #4: ffffffffa489d660 (printk_legacy_map-wait-type-override){....}-{3:3}, at: console_emit_next_record+0x1df/0x4c0 [ 3.557528][ T74] irq event stamp: 2231504 [ 3.557529][ T74] hardirqs last enabled at (2231503): [] __down_trylock_console_sem+0x86/0xa0 [ 3.557532][ T74] hardirqs last disabled at (2231504): [] console_emit_next_record+0x3d4/0x4c0 [ 3.557534][ T74] softirqs last enabled at (2231360): [] handle_softirqs+0x67c/0x900 [ 3.557537][ T74] softirqs last disabled at (2231355): [] __irq_exit_rcu+0x145/0x1c0 [ 3.557539][ T74] Preemption disabled at: [ 3.557540][ T74] [<0000000000000000>] 0x0 [ 3.557546][ T74] CPU: 1 UID: 0 PID: 74 Comm: systemd-tmpfile Tainted: G W 7.2.0-virtme #1 PREEMPT(full) [ 3.557550][ T74] Tainted: [W]=WARN [ 3.557551][ T74] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 3.557552][ T74] Call Trace: [ 3.557554][ T74] [ 3.557555][ T74] dump_stack_lvl+0x6f/0xa0 [ 3.557561][ T74] __might_resched.cold+0x1fe/0x2c1 [ 3.557566][ T74] ? perf_trace_sched_switch+0x7c0/0x7c0 [ 3.557570][ T74] ? __kmalloc_noprof+0xdb/0x760 [ 3.557575][ T74] __kmalloc_noprof+0x443/0x760 [ 3.557577][ T74] ? alloc_buf.isra.0+0x4b/0x260 [ 3.557583][ T74] ? do_raw_spin_unlock+0x59/0x250 [ 3.557586][ T74] alloc_buf.isra.0+0x4b/0x260 [ 3.557590][ T74] put_chars+0x1e1/0x2f0 [ 3.557592][ T74] ? __send_to_port+0x420/0x420 [ 3.557593][ T74] ? rcu_lockdep_current_cpu_online+0x3f/0x1b0 [ 3.557597][ T74] ? validate_chain+0x38b/0xc20 [ 3.557602][ T74] hvc_console_print+0x292/0x780 [ 3.557609][ T74] ? hvc_write+0x3a0/0x3a0 [ 3.557612][ T74] ? rcu_is_watching+0x16/0xd0 [ 3.557614][ T74] ? lock_acquire+0x13c/0x160 [ 3.557618][ T74] console_emit_next_record+0x22f/0x4c0 [ 3.557622][ T74] ? devkmsg_read+0x4b0/0x4b0 [ 3.557624][ T74] ? console_flush_one_record+0x106/0x710 [ 3.557627][ T74] ? rcu_is_watching+0x16/0xd0 [ 3.557628][ T74] ? lock_acquire+0x13c/0x160 [ 3.557632][ T74] console_flush_one_record+0x46f/0x710 [ 3.557636][ T74] ? console_emit_next_record+0x4c0/0x4c0 [ 3.557638][ T74] ? __lock_acquire+0x518/0xc20 [ 3.557643][ T74] console_unlock+0xee/0x1f0 [ 3.557646][ T74] ? console_flush_one_record+0x710/0x710 [ 3.557648][ T74] ? rcu_is_watching+0x16/0xd0 [ 3.557650][ T74] ? lock_acquire+0xe0/0x160 [ 3.557653][ T74] ? __down_trylock_console_sem+0x5e/0xa0 [ 3.557655][ T74] ? vprintk_emit+0x320/0x3e0 [ 3.557658][ T74] vprintk_emit+0x37c/0x3e0 [ 3.557661][ T74] ? wake_up_klogd_work_func+0x90/0x90 [ 3.557664][ T74] ? __lock_acquire+0x518/0xc20 [ 3.557667][ T74] _printk+0xc7/0x100 [ 3.557671][ T74] ? snapshot_read.cold+0x21/0x21 [ 3.557674][ T74] ? do_raw_spin_lock+0x131/0x280 [ 3.557676][ T74] ? __rwlock_init+0x150/0x150 [ 3.557680][ T74] ? do_raw_spin_lock+0x131/0x280 [ 3.557683][ T74] do_exit.cold+0x82/0x9c [ 3.557686][ T74] ? exit_notify+0x890/0x890 [ 3.557688][ T74] ? __lock_release.isra.0+0x69/0x1a0 [ 3.557690][ T74] ? rcu_is_watching+0x16/0xd0 [ 3.557694][ T74] do_group_exit+0xb8/0x370 [ 3.557697][ T74] __x64_sys_exit_group+0x3c/0x50 [ 3.557698][ T74] x64_sys_call+0x1567/0x1570 [ 3.557701][ T74] do_syscall_64+0xff/0x530 [ 3.557705][ T74] ? exc_page_fault+0xee/0x100 [ 3.557708][ T74] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 3.557710][ T74] RIP: 0033:0x7ff2c84e41b8 [ 3.557712][ T74] Code: Unable to access opcode bytes at 0x7ff2c84e418e. [ 3.557713][ T74] RSP: 002b:00007ffdfabbada8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7 [ 3.557715][ T74] RAX: ffffffffffffffda RBX: 00007ff2c8614f88 RCX: 00007ff2c84e41b8 [ 3.557716][ T74] RDX: 00007ff2c7d424c8 RSI: fffffffffffffe90 RDI: 0000000000000049 [ 3.557717][ T74] RBP: 00007ffdfabbae00 R08: 0000000000000000 R09: 0000000000001000 [ 3.557718][ T74] R10: 00007ffdfabbabc0 R11: 0000000000000246 R12: 0000000000000001 [ 3.557719][ T74] R13: 0000000000000049 R14: 00007ff2c8613680 R15: 00007ff2c8614fa0 [ 3.557722][ T1] virtme-ng-init: Failed to read '/usr/lib/tmpfiles.d/audit.conf': Permission denied [ 3.557722][ T1] Failed to create directory or subvolume "/var/spool/at/spool": Permission denied [ 3.557722][ T1] Failed to create file /var/spool/at/.SEQ: Permission denied [ 3.557722][ T1] Failed to opendir() '/proc/self/fd/6': Permission denied [ 3.557726][ T74] [ 3.573925][ T1] virtme-ng-init: basic initialization done [ 3.623622][ T80] ip (80) used greatest stack depth: 24632 bytes left [ 3.638451][ T76] virtme-ng-init: Starting systemd-udevd version 259.8-1.fc44 [ 3.638918][ T76] virtme-ng-init: triggering udev coldplug [ 6.001016][ T76] virtme-ng-init: waiting for udev to settle [ 6.001030][ T76] BUG: sleeping function called from invalid context at ./include/linux/sched/mm.h:322 [ 6.001033][ T76] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 76, name: virtme-ng-init [ 6.001034][ T76] preempt_count: 1, expected: 0 [ 6.001035][ T76] RCU nest depth: 0, expected: 0 [ 6.001041][ T76] locks held by virtme-ng-init/76: 4, last CPU#2: [ 6.001043][ T76] #0: ffffffffa497ddc0 (console_lock){+.+.}-{0:0}, at: vprintk_emit+0x320/0x3e0 [ 6.001055][ T76] #1: ffffffffa497de38 (console_srcu){....}-{0:0}, at: console_flush_one_record+0x106/0x710 [ 6.001059][ T76] #2: ffffffffa489d760 (console_owner){....}-{0:0}, at: console_lock_spinning_enable+0x40/0x60 [ 6.001064][ T76] #3: ffffffffa489d660 (printk_legacy_map-wait-type-override){....}-{3:3}, at: console_emit_next_record+0x1df/0x4c0 [ 6.001068][ T76] irq event stamp: 4496 [ 6.001069][ T76] hardirqs last enabled at (4495): [] __down_trylock_console_sem+0x86/0xa0 [ 6.001071][ T76] hardirqs last disabled at (4496): [] console_emit_next_record+0x3d4/0x4c0 [ 6.001074][ T76] softirqs last enabled at (4192): [] handle_softirqs+0x67c/0x900 [ 6.001077][ T76] softirqs last disabled at (2791): [] __irq_exit_rcu+0x145/0x1c0 [ 6.001079][ T76] Preemption disabled at: [ 6.001080][ T76] [] vprintk_emit+0x31b/0x3e0 [ 6.001085][ T76] CPU: 2 UID: 0 PID: 76 Comm: virtme-ng-init Tainted: G W 7.2.0-virtme #1 PREEMPT(full) [ 6.001088][ T76] Tainted: [W]=WARN [ 6.001089][ T76] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 6.001090][ T76] Call Trace: [ 6.001092][ T76] [ 6.001094][ T76] dump_stack_lvl+0x6f/0xa0 [ 6.001099][ T76] ? vprintk_emit+0x31b/0x3e0 [ 6.001101][ T76] __might_resched.cold+0x1fe/0x2c1 [ 6.001105][ T76] ? perf_trace_sched_switch+0x7c0/0x7c0 [ 6.001109][ T76] ? __kmalloc_noprof+0xdb/0x760 [ 6.001113][ T76] __kmalloc_noprof+0x443/0x760 [ 6.001115][ T76] ? alloc_buf.isra.0+0x4b/0x260 [ 6.001122][ T76] ? do_raw_spin_unlock+0x59/0x250 [ 6.001125][ T76] alloc_buf.isra.0+0x4b/0x260 [ 6.001128][ T76] put_chars+0x1e1/0x2f0 [ 6.001131][ T76] ? __send_to_port+0x420/0x420 [ 6.001133][ T76] ? validate_chain+0x38b/0xc20 [ 6.001138][ T76] hvc_console_print+0x292/0x780 [ 6.001146][ T76] ? hvc_write+0x3a0/0x3a0 [ 6.001148][ T76] ? rcu_is_watching+0x16/0xd0 [ 6.001151][ T76] ? lock_acquire+0x13c/0x160 [ 6.001155][ T76] console_emit_next_record+0x22f/0x4c0 [ 6.001159][ T76] ? devkmsg_read+0x4b0/0x4b0 [ 6.001160][ T76] ? console_flush_one_record+0x106/0x710 [ 6.001163][ T76] ? rcu_is_watching+0x16/0xd0 [ 6.001165][ T76] ? lock_acquire+0x13c/0x160 [ 6.001169][ T76] console_flush_one_record+0x46f/0x710 [ 6.001173][ T76] ? console_emit_next_record+0x4c0/0x4c0 [ 6.001174][ T76] ? __lock_acquire+0x518/0xc20 [ 6.001180][ T76] console_unlock+0xee/0x1f0 [ 6.001182][ T76] ? console_flush_one_record+0x710/0x710 [ 6.001184][ T76] ? rcu_is_watching+0x16/0xd0 [ 6.001186][ T76] ? lock_acquire+0xe0/0x160 [ 6.001189][ T76] ? __down_trylock_console_sem+0x5e/0xa0 [ 6.001191][ T76] ? vprintk_emit+0x320/0x3e0 [ 6.001193][ T76] vprintk_emit+0x37c/0x3e0 [ 6.001196][ T76] ? wake_up_klogd_work_func+0x90/0x90 [ 6.001199][ T76] ? _copy_from_iter+0x1bb/0x1810 [ 6.001204][ T76] devkmsg_emit.constprop.0+0xbc/0xf1 [ 6.001208][ T76] ? vprintk_emit.cold+0x107/0x107 [ 6.001209][ T76] ? simple_strntoull+0x10f/0x140 [ 6.001213][ T76] ? date_str+0x1e0/0x1e0 [ 6.001216][ T76] ? devkmsg_write+0xd1/0x2c0 [ 6.001219][ T76] devkmsg_write.cold+0x5a/0x8b [ 6.001222][ T76] ? vprintk_default+0x20/0x20 [ 6.001226][ T76] ? vprintk_default+0x20/0x20 [ 6.001227][ T76] new_sync_write+0x33e/0x760 [ 6.001231][ T76] ? kasan_quarantine_put+0x62/0x2b0 [ 6.001235][ T76] ? new_sync_read+0x750/0x750 [ 6.001238][ T76] ? lock_acquire.part.0+0xd4/0x280 [ 6.001242][ T76] ? __lock_release.isra.0+0x69/0x1a0 [ 6.001246][ T76] ? __fget_files+0x1e3/0x460 [ 6.001251][ T76] vfs_write+0x655/0x970 [ 6.001254][ T76] ksys_write+0x116/0x250 [ 6.001257][ T76] ? __ia32_sys_read+0xc0/0xc0 [ 6.001259][ T76] ? rcu_is_watching+0x16/0xd0 [ 6.001262][ T76] do_syscall_64+0xff/0x530 [ 6.001265][ T76] ? exc_page_fault+0xee/0x100 [ 6.001268][ T76] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 6.001270][ T76] RIP: 0033:0x7f2cbc7abed2 [ 6.001273][ T76] Code: 08 0f 85 51 ec ff ff 49 89 fb 48 89 f0 48 89 d7 48 89 ce 4c 89 c2 4d 89 ca 4c 8b 44 24 08 4c 8b 4c 24 10 4c 89 5c 24 08 0f 05 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 f3 0f 1e fa 55 48 89 e5 [ 6.001275][ T76] RSP: 002b:00007f2cbc6f2c68 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 6.001278][ T76] RAX: ffffffffffffffda RBX: 000000000000002e RCX: 00007f2cbc7abed2 [ 6.001279][ T76] RDX: 000000000000002e RSI: 00007f2cb80012c0 RDI: 0000000000000003 [ 6.001280][ T76] RBP: 00007f2cbc6f2c90 R08: 0000000000000000 R09: 0000000000000000 [ 6.001281][ T76] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f2cbc8360a0 [ 6.001281][ T76] R13: 00007f2cbc7343e0 R14: 00007f2cb80012c0 R15: 00007f2cbc6f2da8 [ 6.001288][ T76] [ 6.024369][ C0] clocksource: Watchdog remote CPU 2 read timed out [ 6.024391][ C0] [ 6.024393][ C0] ======================================================== [ 6.024393][ C0] WARNING: possible irq lock inversion dependency detected [ 6.024395][ C0] 7.2.0-virtme #1 Tainted: G W [ 6.024396][ C0] -------------------------------------------------------- [ 6.024397][ C0] (udev-worker)/92 just changed the state of lock: [ 6.024398][ C0] ffffffffa489d760 (console_owner){..-.}-{0:0}, at: console_trylock_spinning+0xa4/0x1e0 [ 6.024408][ C0] but this lock took another, SOFTIRQ-unsafe lock in the past: [ 6.024409][ C0] (fs_reclaim){+.+.}-{0:0} [ 6.024410][ C0] [ 6.024410][ C0] [ 6.024410][ C0] and interrupts could create inverse lock ordering between them. [ 6.024410][ C0] [ 6.024411][ C0] [ 6.024411][ C0] other info that might help us debug this: [ 6.024412][ C0] Possible interrupt unsafe locking scenario: [ 6.024412][ C0] [ 6.024412][ C0] CPU0 CPU1 [ 6.024413][ C0] ---- ---- [ 6.024413][ C0] lock(fs_reclaim); [ 6.024414][ C0] local_irq_disable(); [ 6.024415][ C0] lock(console_owner); [ 6.024416][ C0] lock(fs_reclaim); [ 6.024417][ C0] [ 6.024417][ C0] lock(console_owner); [ 6.024418][ C0] [ 6.024418][ C0] *** DEADLOCK *** [ 6.024418][ C0] [ 6.024418][ C0] locks held by (udev-worker)/92: 2, last CPU#0: [ 6.024419][ C0] #0: ffa0000000007c90 ((&watchdog_timer)){+.-.}-{0:0}, at: call_timer_fn+0x110/0x4d0 [ 6.024425][ C0] #1: ffffffffa49e29f8 (watchdog_lock){+.-.}-{3:3}, at: clocksource_watchdog+0x15/0x30 [ 6.024429][ C0] [ 6.024429][ C0] the shortest dependencies between 2nd lock and 1st lock: [ 6.024433][ C0] -> (fs_reclaim){+.+.}-{0:0} { [ 6.024435][ C0] HARDIRQ-ON-W at: [ 6.024436][ C0] __lock_acquire+0x388/0xc20 [ 6.024439][ C0] lock_acquire.part.0+0xd4/0x280 [ 6.024441][ C0] fs_reclaim_acquire+0xd5/0x120 [ 6.024444][ C0] __kmalloc_cache_noprof+0x6e/0x620 [ 6.024445][ C0] kthread_create_worker_on_node+0xea/0x210 [ 6.024448][ C0] workqueue_init+0x2a/0x680 [ 6.024452][ C0] kernel_init_freeable+0x2fe/0x630 [ 6.024455][ C0] kernel_init+0x21/0x150 [ 6.024459][ C0] ret_from_fork+0x474/0x6b0 [ 6.024462][ C0] ret_from_fork_asm+0x11/0x20 [ 6.024465][ C0] SOFTIRQ-ON-W at: [ 6.024465][ C0] __lock_acquire+0x388/0xc20 [ 6.024467][ C0] lock_acquire.part.0+0xd4/0x280 [ 6.024468][ C0] fs_reclaim_acquire+0xd5/0x120 [ 6.024470][ C0] __kmalloc_cache_noprof+0x6e/0x620 [ 6.024471][ C0] kthread_create_worker_on_node+0xea/0x210 [ 6.024472][ C0] workqueue_init+0x2a/0x680 [ 6.024474][ C0] kernel_init_freeable+0x2fe/0x630 [ 6.024475][ C0] kernel_init+0x21/0x150 [ 6.024476][ C0] ret_from_fork+0x474/0x6b0 [ 6.024478][ C0] ret_from_fork_asm+0x11/0x20 [ 6.024479][ C0] INITIAL USE at: [ 6.024480][ C0] __lock_acquire+0x388/0xc20 [ 6.024481][ C0] lock_acquire.part.0+0xd4/0x280 [ 6.024483][ C0] fs_reclaim_acquire+0xd5/0x120 [ 6.024484][ C0] __kmalloc_cache_noprof+0x6e/0x620 [ 6.024485][ C0] kthread_create_worker_on_node+0xea/0x210 [ 6.024487][ C0] workqueue_init+0x2a/0x680 [ 6.024488][ C0] kernel_init_freeable+0x2fe/0x630 [ 6.024490][ C0] kernel_init+0x21/0x150 [ 6.024491][ C0] ret_from_fork+0x474/0x6b0 [ 6.024492][ C0] ret_from_fork_asm+0x11/0x20 [ 6.024494][ C0] } [ 6.024494][ C0] ... key at: [] __fs_reclaim_map+0x0/0xbe0 [ 6.024498][ C0] ... acquired at: [ 6.024499][ C0] __lock_acquire+0x518/0xc20 [ 6.024500][ C0] lock_acquire.part.0+0xd4/0x280 [ 6.024502][ C0] fs_reclaim_acquire+0xd5/0x120 [ 6.024503][ C0] __kmalloc_noprof+0xd3/0x760 [ 6.024504][ C0] alloc_buf.isra.0+0x4b/0x260 [ 6.024508][ C0] put_chars+0x1e1/0x2f0 [ 6.024509][ C0] hvc_console_print+0x292/0x780 [ 6.024511][ C0] console_emit_next_record+0x22f/0x4c0 [ 6.024513][ C0] console_flush_one_record+0x46f/0x710 [ 6.024515][ C0] console_unlock+0xee/0x1f0 [ 6.024516][ C0] vprintk_emit+0x37c/0x3e0 [ 6.024518][ C0] _printk+0xc7/0x100 [ 6.024520][ C0] tun_init+0x24/0xb0 [ 6.024523][ C0] do_one_initcall+0x124/0x4f0 [ 6.024524][ C0] kernel_init_freeable+0x596/0x630 [ 6.024526][ C0] kernel_init+0x21/0x150 [ 6.024527][ C0] ret_from_fork+0x474/0x6b0 [ 6.024528][ C0] ret_from_fork_asm+0x11/0x20 [ 6.024530][ C0] [ 6.024530][ C0] -> (console_owner){..-.}-{0:0} { [ 6.024532][ C0] IN-SOFTIRQ-W at: [ 6.024532][ C0] __lock_acquire+0x388/0xc20 [ 6.024534][ C0] lock_acquire.part.0+0xd4/0x280 [ 6.024535][ C0] console_trylock_spinning+0xb5/0x1e0 [ 6.024537][ C0] vprintk_emit+0x320/0x3e0 [ 6.024537][ C0] _printk+0xc7/0x100 [ 6.024539][ C0] clocksource_watchdog.part.0.cold+0x105/0x1dd [ 6.024541][ C0] call_timer_fn+0x160/0x4d0 [ 6.024543][ C0] __run_timers+0x68f/0xaa0 [ 6.024545][ C0] run_timer_softirq+0xf0/0x160 [ 6.024546][ C0] handle_softirqs+0x1d3/0x900 [ 6.024549][ C0] __irq_exit_rcu+0x145/0x1c0 [ 6.024550][ C0] irq_exit_rcu+0xe/0x30 [ 6.024551][ C0] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 6.024553][ C0] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6.024554][ C0] check_preemption_disabled+0x11/0xe0 [ 6.024556][ C0] rcu_is_watching+0x16/0xd0 [ 6.024557][ C0] kernel_text_address+0x88/0x170 [ 6.024559][ C0] __kernel_text_address+0x12/0x30 [ 6.024560][ C0] unwind_get_return_address+0x67/0xd0 [ 6.024563][ C0] arch_stack_walk+0xed/0x130 [ 6.024564][ C0] stack_trace_save+0x93/0xc0 [ 6.024565][ C0] kasan_save_stack+0x2f/0x50 [ 6.024567][ C0] kasan_save_track+0x14/0x30 [ 6.024568][ C0] kasan_save_free_info+0x3b/0x60 [ 6.024570][ C0] __kasan_slab_free+0x43/0x70 [ 6.024571][ C0] kmem_cache_free+0xf8/0x550 [ 6.024572][ C0] do_sys_openat2+0xff/0x170 [ 6.024574][ C0] __x64_sys_openat+0x10e/0x210 [ 6.024575][ C0] do_syscall_64+0xff/0x530 [ 6.024577][ C0] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 6.024578][ C0] INITIAL USE at: [ 6.024579][ C0] } [ 6.024579][ C0] ... key at: [] console_owner_dep_map+0x0/0x60 [ 6.024582][ C0] ... acquired at: [ 6.024582][ C0] mark_lock+0x1d7/0xa00 [ 6.024584][ C0] mark_usage+0x42/0x170 [ 6.024585][ C0] __lock_acquire+0x388/0xc20 [ 6.024586][ C0] lock_acquire.part.0+0xd4/0x280 [ 6.024588][ C0] console_trylock_spinning+0xb5/0x1e0 [ 6.024589][ C0] vprintk_emit+0x320/0x3e0 [ 6.024590][ C0] _printk+0xc7/0x100 [ 6.024591][ C0] clocksource_watchdog.part.0.cold+0x105/0x1dd [ 6.024592][ C0] call_timer_fn+0x160/0x4d0 [ 6.024594][ C0] __run_timers+0x68f/0xaa0 [ 6.024595][ C0] run_timer_softirq+0xf0/0x160 [ 6.024597][ C0] handle_softirqs+0x1d3/0x900 [ 6.024598][ C0] __irq_exit_rcu+0x145/0x1c0 [ 6.024599][ C0] irq_exit_rcu+0xe/0x30 [ 6.024600][ C0] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 6.024602][ C0] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6.024603][ C0] check_preemption_disabled+0x11/0xe0 [ 6.024604][ C0] rcu_is_watching+0x16/0xd0 [ 6.024605][ C0] kernel_text_address+0x88/0x170 [ 6.024606][ C0] __kernel_text_address+0x12/0x30 [ 6.024607][ C0] unwind_get_return_address+0x67/0xd0 [ 6.024608][ C0] arch_stack_walk+0xed/0x130 [ 6.024609][ C0] stack_trace_save+0x93/0xc0 [ 6.024610][ C0] kasan_save_stack+0x2f/0x50 [ 6.024611][ C0] kasan_save_track+0x14/0x30 [ 6.024612][ C0] kasan_save_free_info+0x3b/0x60 [ 6.024613][ C0] __kasan_slab_free+0x43/0x70 [ 6.024614][ C0] kmem_cache_free+0xf8/0x550 [ 6.024615][ C0] do_sys_openat2+0xff/0x170 [ 6.024616][ C0] __x64_sys_openat+0x10e/0x210 [ 6.024617][ C0] do_syscall_64+0xff/0x530 [ 6.024619][ C0] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 6.024620][ C0] [ 6.024620][ C0] [ 6.024620][ C0] stack backtrace: [ 6.024622][ C0] CPU: 0 UID: 0 PID: 92 Comm: (udev-worker) Tainted: G W 7.2.0-virtme #1 PREEMPT(full) [ 6.024625][ C0] Tainted: [W]=WARN [ 6.024626][ C0] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 6.024627][ C0] Call Trace: [ 6.024628][ C0] [ 6.024629][ C0] dump_stack_lvl+0x6f/0xa0 [ 6.024633][ C0] print_irq_inversion_bug.part.0.cold+0xe6/0x143 [ 6.024636][ C0] mark_lock_irq+0x989/0x9c0 [ 6.024637][ C0] ? add_lock_to_list+0x2c/0x1a0 [ 6.024640][ C0] mark_lock+0x1d7/0xa00 [ 6.024642][ C0] mark_usage+0x42/0x170 [ 6.024644][ C0] __lock_acquire+0x388/0xc20 [ 6.024646][ C0] lock_acquire.part.0+0xd4/0x280 [ 6.024648][ C0] ? console_trylock_spinning+0xa4/0x1e0 [ 6.024650][ C0] ? rcu_is_watching+0x16/0xd0 [ 6.024651][ C0] ? lock_acquire+0x13c/0x160 [ 6.024653][ C0] console_trylock_spinning+0xb5/0x1e0 [ 6.024655][ C0] ? console_trylock_spinning+0xa4/0x1e0 [ 6.024656][ C0] vprintk_emit+0x320/0x3e0 [ 6.024658][ C0] ? wake_up_klogd_work_func+0x90/0x90 [ 6.024660][ C0] ? clocksource_watchdog.part.0+0x450/0x540 [ 6.024662][ C0] _printk+0xc7/0x100 [ 6.024663][ C0] ? snapshot_read.cold+0x21/0x21 [ 6.024665][ C0] ? clocksource_watchdog.part.0+0x450/0x540 [ 6.024667][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 6.024669][ C0] clocksource_watchdog.part.0.cold+0x105/0x1dd [ 6.024671][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 6.024672][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 6.024674][ C0] call_timer_fn+0x160/0x4d0 [ 6.024676][ C0] ? detach_if_pending+0x1d0/0x1d0 [ 6.024678][ C0] ? debug_object_active_state+0x430/0x430 [ 6.024681][ C0] ? find_held_lock+0x2b/0x80 [ 6.024683][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 6.024685][ C0] ? rcu_is_watching+0x16/0xd0 [ 6.024686][ C0] __run_timers+0x68f/0xaa0 [ 6.024688][ C0] ? clocksource_watchdog.part.0+0x540/0x540 [ 6.024690][ C0] ? __bpf_trace_itimer_expire+0x10/0x10 [ 6.024692][ C0] ? __lock_acquire+0x518/0xc20 [ 6.024695][ C0] ? __rwlock_init+0x150/0x150 [ 6.024698][ C0] run_timer_softirq+0xf0/0x160 [ 6.024700][ C0] ? __run_timers+0xaa0/0xaa0 [ 6.024701][ C0] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 6.024704][ C0] ? rcu_is_watching+0x16/0xd0 [ 6.024705][ C0] handle_softirqs+0x1d3/0x900 [ 6.024707][ C0] ? __lock_release.isra.0+0x69/0x1a0 [ 6.024709][ C0] ? _local_bh_enable+0xc0/0xc0 [ 6.024711][ C0] __irq_exit_rcu+0x145/0x1c0 [ 6.024713][ C0] irq_exit_rcu+0xe/0x30 [ 6.024714][ C0] sysvec_apic_timer_interrupt+0x9d/0xe0 [ 6.024716][ C0] [ 6.024717][ C0] [ 6.024717][ C0] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6.024719][ C0] RIP: 0010:check_preemption_disabled+0x11/0xe0 [ 6.024721][ C0] Code: 75 0f 65 8b 05 84 69 68 02 85 c0 74 04 90 0f 0b 90 e9 93 fb ff ff 0f 1f 00 65 8b 15 21 27 68 02 65 44 8b 15 29 27 68 02 85 d2 <74> 04 44 89 d0 c3 9c 58 f6 c4 02 74 f5 65 48 8b 15 fa 26 68 02 f6 [ 6.024723][ C0] RSP: 0018:ffa00000005d7970 EFLAGS: 00000202 [ 6.024725][ C0] RAX: 0000000000000001 RBX: 00007fe07b7e254e RCX: 1ffffffff4a69ef0 [ 6.024726][ C0] RDX: 0000000000000001 RSI: ffffffffa3894960 RDI: ffffffffa38949a0 [ 6.024727][ C0] RBP: ffa00000005d7a68 R08: ffa00000005d7a30 R09: ffa00000005d7958 [ 6.024728][ C0] R10: 0000000000000000 R11: 0000000000000001 R12: ffa00000005d7a98 [ 6.024729][ C0] R13: 0000000000000000 R14: ff110000122b0040 R15: ffffffffa162588f [ 6.024729][ C0] ? do_sys_openat2+0xff/0x170 [ 6.024732][ C0] rcu_is_watching+0x16/0xd0 [ 6.024733][ C0] ? __unwind_start+0x42e/0x900 [ 6.024734][ C0] kernel_text_address+0x88/0x170 [ 6.024736][ C0] __kernel_text_address+0x12/0x30 [ 6.024737][ C0] ? write_profile+0xf0/0xf0 [ 6.024739][ C0] unwind_get_return_address+0x67/0xd0 [ 6.024740][ C0] arch_stack_walk+0xed/0x130 [ 6.024743][ C0] stack_trace_save+0x93/0xc0 [ 6.024744][ C0] ? filter_irq_stacks+0xd0/0xd0 [ 6.024746][ C0] ? do_sys_openat2+0xff/0x170 [ 6.024748][ C0] kasan_save_stack+0x2f/0x50 [ 6.024749][ C0] ? kasan_save_stack+0x2f/0x50 [ 6.024750][ C0] ? kasan_save_track+0x14/0x30 [ 6.024751][ C0] ? kasan_save_free_info+0x3b/0x60 [ 6.024752][ C0] ? __kasan_slab_free+0x43/0x70 [ 6.024753][ C0] ? kmem_cache_free+0xf8/0x550 [ 6.024754][ C0] ? do_sys_openat2+0xff/0x170 [ 6.024755][ C0] ? __x64_sys_openat+0x10e/0x210 [ 6.024756][ C0] ? do_syscall_64+0xff/0x530 [ 6.024758][ C0] ? entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 6.024762][ C0] kasan_save_track+0x14/0x30 [ 6.024764][ C0] kasan_save_free_info+0x3b/0x60 [ 6.024765][ C0] __kasan_slab_free+0x43/0x70 [ 6.024767][ C0] kmem_cache_free+0xf8/0x550 [ 6.024768][ C0] do_sys_openat2+0xff/0x170 [ 6.024770][ C0] ? build_open_flags+0x6e0/0x6e0 [ 6.024771][ C0] ? rcu_is_watching+0x16/0xd0 [ 6.024772][ C0] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 6.024774][ C0] ? lockdep_hardirqs_on+0x91/0x130 [ 6.024776][ C0] __x64_sys_openat+0x10e/0x210 [ 6.024777][ C0] ? fput_close_sync+0xde/0x1b0 [ 6.024779][ C0] ? __ia32_sys_open+0x1b0/0x1b0 [ 6.024781][ C0] ? rcu_is_watching+0x16/0xd0 [ 6.024782][ C0] ? trace_irq_enable.constprop.0+0x9b/0x160 [ 6.024784][ C0] do_syscall_64+0xff/0x530 [ 6.024785][ C0] ? exc_page_fault+0xee/0x100 [ 6.024787][ C0] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 6.024788][ C0] RIP: 0033:0x7fe07b7e254e [ 6.024790][ C0] Code: 4d 89 d8 e8 b4 bd 00 00 4c 8b 5d f8 41 8b 93 08 03 00 00 59 5e 48 83 f8 fc 74 11 c9 c3 0f 1f 80 00 00 00 00 48 8b 45 10 0f 05 c3 83 e2 39 83 fa 08 75 e7 e8 03 ff ff ff 0f 1f 00 f3 0f 1e fa [ 6.024791][ C0] RSP: 002b:00007fffc46712b0 EFLAGS: 00000202 ORIG_RAX: 0000000000000101 [ 6.024793][ C0] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fe07b7e254e [ 6.024793][ C0] RDX: 00000000002a0000 RSI: 00005614673ec460 RDI: 0000000000000027 [ 6.024794][ C0] RBP: 00007fffc46712c0 R08: 0000000000000000 R09: 0000000000000000 [ 6.024795][ C0] R10: 0000000000000000 R11: 0000000000000202 R12: 00005614673edd21 [ 6.024796][ C0] R13: 0000000000000027 R14: 00005614673ec460 R15: 0000000000000026 [ 6.024798][ C0] [ 6.390674][ T76] virtme-ng-init: udev is done [ 6.397393][ T1] virtme-ng-init: initialization done