mirror of
git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs.git
synced 2026-10-28 12:53:14 +01:00
dev
2845
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Sean Christopherson
|
41e005e2c5 |
KVM: WARN if vCPU creation is in-progress when locking all vCPUs
Now that KVM holds kvm->lock for the entirety of vCPU creation, from when created_vcpus is incremented until the new vCPU is fully onlined, WARN if the impossible happens and KVM somehow sees a discrepancy between the number of vCPUs "created" and "onlined". Signed-off-by: Sean Christopherson <[email protected]> Tested-by: Jean-Christophe Guillain <[email protected]> Tested-by: Naveen N Rao (AMD) <[email protected]> Message-ID: <[email protected]> Signed-off-by: Paolo Bonzini <[email protected]> |
||
|
Sean Christopherson
|
700f8d2443 |
Revert "KVM: Check for duplicate vcpu_id as early as possible"
Now that KVM uses kvm_get_vcpu_by_id() to check for an existing vCPU ID before doing any meaningful work, which was made possible by holding kvm->lock for the entirety of vCPU creation, revert the now-redundant "early" vCPU ID tracking. The claims about the impact of kvm->vcpu_ids on the memory footprint were a wee bit wrong: the worst case scenario isn't 256 bytes per VM, it's 256 "unsigned longs" per VM, i.e. 2048 bytes per VM. Increasing the size of "struct kvm" by 2048 nearly doubled the total size on many architectures, and tripped x86's KVM_SANITY_CHECK_VM_STRUCT_SIZE, which was added to detect this *exact* scenario, where a single change significantly increased the size of "struct kvm". I.e. attempting to build KVM with CONFIG_DEBUG_KERNEL=n fails on x86 (the build failures got missed because all build bots apparently test only CONFIG_DEBUG_KERNEL=y kernels, and maintainers' test flows were similarly lacking). This reverts commit |
||
|
Sean Christopherson
|
c5866d42f7 |
KVM: Move check for existing vCPU ID to the top of vCPU creation
Now that kvm->lock is held for the entirety of vCPU creation, check for a
conflicting vCPU ID at the begnning of vCPU creation, before the arch
precreate() hook is invoked. This will allow reverting commit
|
||
|
Sean Christopherson
|
6418b715d5 |
KVM: Protect all of kvm_vm_ioctl_create_vcpu() with kvm->lock
When creating a vCPU, don't drop kvm->lock to when doing the bulk of actual
vCPU creation, as allowing multiple vCPUs to be created in parallel adds
significant complexity in KVM (as evidenced by the many related bugs), and
all known VMMs fully serialize vCPU creation. Remove all manually locking
of kvm->lock from kvm_arch_vcpu_{post,}create() for obvious reasons.
For many years, "everyone" has assumed that dropping kvm->lock was done for
performance reasons optimization, e.g. to allow userspace to create all
vCPUs concurrently for latency purposes. But as above, no known VMM does
that. Looking at the history of this code, before commit
|
||
|
Sean Christopherson
|
8264c0d73f |
KVM: Reject attempts to lock all vCPUs if vCPU creation is in-progress
Reject locking of all vCPUs if vCPU creation is in-progress, i.e. if the number of "created" vCPUs doesn't match the number of "onlined" vCPUs. It's simply not possible to guarantee that KVM has truly locked all vCPUs if one or more vCPUs are actively being created. Holding kvm->lock does prevent in-flight vCPUs from being fully onlined, but it's infeasible for common KVM to know whether or not that provides sufficient protection. In practice, this is likely a minor bug fix for the ARM and RISC-V usage of kvm_trylock_all_vcpus(), and a glorified nop for everything else. E.g. ARM's kvm_timer_vcpu_init() can race kvm_vm_ioctl_set_counter_offset() with respect to observing KVM_ARCH_FLAG_VM_COUNTER_OFFSET. Opportunistically drop x86's existing manual checks on vCPU creation being in-progress as all of x86's checks immediately precede or follow locking of all vCPUs. Leave arm64 and RISC-V alone for the moment, as their checks aren't as obviously redundant/equivalent. Signed-off-by: Sean Christopherson <[email protected]> Tested-by: Jean-Christophe Guillain <[email protected]> Tested-by: Naveen N Rao (AMD) <[email protected]> Message-ID: <[email protected]> Signed-off-by: Paolo Bonzini <[email protected]> |
||
|
Zeng Chi
|
119e9db233 |
KVM: Don't pre-reserve xarray entries when storing empty/NULL attributes
Skip the xarray reservation loop when clearing all memory attributes, as storing NULL only erases the entry and never needs to allocate, so no reservation (and no cleanup of a failed one) is required in that case. Suggested-by: Sean Christopherson <[email protected]> Cc: David Ballesteros <[email protected]> Signed-off-by: Zeng Chi <[email protected]> Link: https://patch.msgid.link/[email protected] [sean: split to separate patch] Signed-off-by: Sean Christopherson <[email protected]> |
||
|
David Ballesteros
|
382e5d514b |
KVM: Ensure memory attributes xarray nodes are accounted to the caller's memcg
Explicitly instantiate the memory attributes xarray with XA_FLAGS_ACCOUNT
to ensure that all allocations are accounted to the memcg. Frustratingly,
memory allocations done in the "fastpath" do not honor the passed in gfp,
even for an explicit xa_reserve(). Only the rare, slow path __xas_nomem()
honors the original gfp. E.g.
xa_reserve(..., GFP_KERNEL_ACCOUNT)
|
-> ...
|
-> __xa_cmpxchg_raw()
|
-> xas_store() <== does not take @gfp
|
-> xas_create()
|
-> xas_alloc()
The bug was confirmed by observing that a process in a cgroup limited to
256 MiB grew radix_tree_node slab by ~512 MiB while its memory.current
stayed near 0.
Fixes:
|
||
|
Zeng Chi
|
277d3623d9 |
KVM: Don't treat reserved xarray entries as having memory attributes
kvm_vm_set_mem_attributes() reserves an xarray entry for every gfn in
the range before storing the new attributes, so that the store loop
can't fail partway through. If one of the reservations fails, e.g. with
-ENOMEM, the entries that were already reserved are left in the array.
That is harmless as far as xa_reserve() is concerned, as the reserved
entries read back as NULL via xa_load(), but it confuses the "does this
range have no attributes at all" check:
if (!attrs)
return !xas_find(&xas, end - 1);
A reserved entry is XA_ZERO_ENTRY, not NULL, and xas_find() returns it
as present. So a leftover reservation makes KVM report that a fully
shared range has attributes even though kvm_get_memory_attributes()
returns none for every gfn in the range. On x86, the next time
mixed-attribute tracking is recomputed for the range (memslot creation,
or a later attribute change that straddles the 2MiB page),
hugepage_has_attrs() treats a fully shared 2MiB range as mixed and
refuses to map it with a hugepage, until userspace happens to set
attributes on the range again.
Drop the shortcut and handle the !attrs case in the per-index loop,
using xas_next_entry() to find the next non-NULL entry. xas_next_entry()
is essentially an optimized xas_find(), so the effective change is that
the !attrs lookup now goes through xas_retry() like the attrs != 0 case,
i.e. reserved entries are skipped and retry entries restart the walk.
Don't check the index when no entry is found, as the xarray leaves the
xas index in a bogus state in that case; no entry simply means the rest
of the range has no attributes.
KVM never stores a non-NULL entry with a value of zero (clearing stores
NULL), but such an entry would be returned by xas_next_entry() and trip
the index check, so WARN if one is ever seen.
Fixes:
|
||
|
Linus Torvalds
|
7bb6284aa7 |
Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm
Pull kvm updates from Paolo Bonzini:
"ARM64:
- Add support for 'slot' based PMU events, paired with new UAPI that
compels the user to select a specific PMU implementation
- Lazy save/restore of vCPU state for pKVM, along with various fixes
and cleanups to the management of vCPU state between the untrusted
host and pKVM hypervisor
- Disable traps of EL1 registers for nested hypervisors when
FEAT_NV2p1 is present, guaranteeing that EL2-specific register bits
are stateful in the EL1 counterpart
- Leverage FEAT_NV3 to avoid unnecessary ERET/TLBI traps when the
scope of those instructions remains 'in host' (i.e. L1
kernel/userspace)
- Pile of fixes for the management of the VNCR pseudo-TLB, such as
under-invalidations and races with concurrent TLBIs on other vCPUs
- Consolidate the non-protected and pKVM view of ICH_VTR_EL2 to a
runtime-patched constant, allowing the same data to be shared with
pKVM prior to dropping host privileges
- Considerable pile of LLM-assisted fixes around the shop but mostly
in the VGIC, our in-kernel generator of bugs (and sometimes
interrupts)
LoongArch:
- Advertise already-supported capabilities
- Some bug fixes about timer and MMIO
- Some hardening about interrupt injection
- Replace kvm_err() with kvm_pr_unimpl()
- Add FPU/LSX/LASX test cases for selftests
RISC-V:
- Svadu/Zicfiss/Zicfilp FWFT support for Guest
- Use try_cmpxchg for IMSIC MRIF RMW
- More arch-specific tracepoints in KVM RISC-V
- Eager page splitting when enabling dirty logging
- Optimize hfence request handling for SMP Guests
- Improve dirty log clearing by skipping zero bits in mask
- Guard HFENCE range loops against overflow
- CPU PM notifiers in KVM RISC-V for non-retentive idle states
- Fix kernel-mode vector context save/restore for Guest
s390:
- Fixes for vfio-ap
- Fixes for the gmap rework
- Fixes for vsie
- AI triggered fixes all over
- diag9c tracing
- code move preparation for the additional arm64 support
- enable CONTEXT_ANALYSIS
x86:
- Perform spring cleaning on x86.{c,h} and asm/kvm_host.h, by adding
regs.c (the kvm_cache_regs.h => regs.h is already applied) and
msrs.{c,h}, and moving relevant code out of x86.c
- Split kvm_mmu in three parts, respectively to describe the format
of page tables, walking the guest page tables and building the page
tables. Always use the same page table walker kvm->arch.gva_walk as
the entry point to convert a guest's virtual address, where the
previous code used two different kvm_mmu structs depending on
whether the walk included nested EPT/NPT or not. Make page fault
vmexits reuse the permission checking machinery that is used for
guest page faults. This is both a cleanup and a baby step towards
supporting XS/XU memory permissions
- Document some of the "fun" gotchas with the APIC base when creating
IRQCHIPs on x86
- Remove a defunct masterclock update from kvm_xen_shared_info_init().
It could result in incorrect kvmclock due to triggering an
unnecessary switch to/from masterclock mode
- Skip Xen runstate time updates if time has effectively gone
backwards, so that the guest doesn't report 100% steal time for
a very, very long time
- Drop KVM's runtime updates of the Xen PV timing CPUID leaf, as KVM
was updating the wrong sub-leaf, and upstream KVM will soon provide
all the information needed by userspace to populate the CPUID field
itself
- Fix a bug where KVM would walk a newly created rmap without holding
the rmap lock (or mmu_lock) during aging
- Fix a bug where aging TDP MMU SPTEs could clobber FROZEN SPTEs
- Fix a variety of #DB priority bugs
- Fix a class of races related to enabling Hyper-V emulation on a
vCPU after the vCPU is visible to the rest of KVM
- Use static calls for nested virtualization ops
- Move more KVM-internal code out of x86's kvm_host.h
- Enumerate support for a variety of Zhaoxin instructions that don't
require explicit virtualization
- Fix missing EFER validation bugs, including in the KVM_SET_SREGS*
path
- Harden kvm_vcpu_map() against double-mapping and thus leaking
references
- Misc fixes and cleanups, e.g. for largely benign syzkaller splats
x86 (Intel):
- Zero a vCPU's entry in VMX's Posted Interrupt Descriptor table used
for IPI virtualization when the vCPU is freed, to fix a
use-after-free where hardware will write to a freed vCPU's PID
- Service local TLB flushes on a failed nested VM-Enter to fix a bug
where KVM could miss a TLB on a future, successful VM-Enter with
the same L2 VPID
- Cap the maximum value shoved into the VMX Preemption Timer to
workaround an erratum that affects all existing Intel CPUs that
support CPUID 0x15
- Fix VPID virtualization bugs where KVM would fail to flush hardware
TLBs
- Harden the TDX "populate" ioctls against bad input, and to prepare
for supporting in-place private<=>shared conversion
x86 (AMD):
- Forcefully invalidate SNP VMSA pages if their backing guest_memfd
page is zapped/invalidated, e.g. due to a PUNCH_HOLE in response to
a Page-State Change request
- Remove a dying VM from the GA Log notifier list before the VM is
actually destroyed, to fix a potential use-after-free
- While FOLL_WRITE was needed in the past to trigger CoW unsharing,
nowadays FOLL_LONGTERM does that already even without FOLL_WRITE,
and in fact, get_user_pages() actually disallows FOLL_WRITE
together with FOLL_LONGTERM. So don't pass FOLL_WRITE when
registering encrypted memory regions, i.e. when pinning SEV/SEV-ES
guest memory, to fix a regression with file-backed memory
introduced by KVM's (correct) usage of long-term pins
(This was reviewed by mm maintainers; for more information, see
commit
|
||
|
Paolo Bonzini
|
d75b484605 |
Merge tag 'kvm-x86-misc-7.3' of https://github.com/kvm-x86/linux into HEAD
KVM x86 misc changes for 7.3 - Fix VPID virtualization bugs where KVM would fail to flush hardware TLBs. - Harden the SNP and TDX "populate" ioctls against bad input, and to prepare for supporting in-place private<=>shared conversion. - Fix a variety of #DB priority bugs. - Fix a class of races related to enabling Hyper-V emulation on a vCPU after the vCPU is visible to the rest of KVM. - Use static calls for nested virtualization ops. - Move more KVM-internal code out of x86's kvm_host.h. - Enumerate support for a variety of Zhaoxin instructions that don't require explicit virtualization. - Fix missing EFER validation bugs, including in the KVM_SET_SREGS* path. - Harden kvm_vcpu_map() against double-mapping and thus leaking references. - Misc fixes and cleanups, e.g. for largely benign syzkaller splats. |
||
|
Paolo Bonzini
|
159ed1ae35 |
Merge tag 'kvm-x86-coco-7.3' of https://github.com/kvm-x86/linux into HEAD
KVM guest_memfd and x86 CoCo changes for 7.3 - Forcefully invalidate SNP VMSA pages if their backing guest_memfd page is zapped/invalidated, e.g. due to a PUNCH_HOLE in response to a Page-State Change request. - Rework the so called "prepare" and "invalidate" guest_memfd hooks to prepare for in-place private<=>shared conversion, and clean up a few warts along the way. |
||
|
Paolo Bonzini
|
9dc54ddece |
Merge tag 'kvm-x86-generic-7.3' of https://github.com/kvm-x86/linux into HEAD
KVM arch-neutral and documentation changes for 7.3 - Remove kvm_debugfs_dir if kvm_init() fails after creating KVM's debugfs. - Document some of the "fun" gotchas with the APIC base when creating IRQCHIPs on x86. - Add a per-VM bitmap to track which vCPU IDs have been "claimed" but for which the vCPU isn't yet online, and use the bitmap to reject duplicate IDs before calling into arch code. This allows arch code to consume vcpu_id without having to worry about cross-vCPU clobbering (at least s390 and x86 have had related bugs). - Zero a vCPU's entry in VMX's Posted Interrupt Descriptor table used for IPI virtualization when the vCPU is freed to fix a use-after-free where hardware will write to a freed vCPU's PID. |
||
|
Paolo Bonzini
|
6486818d53 |
Merge tag 'kvm-x86-maintainers-7.3' of https://github.com/kvm-x86/linux into HEAD
KVM MAINTAINERS changes for 7.3 - Add the kvm-x86 tree to KVM x86 entries so that humans and robots alike can more easily find in-flight x86 changes. - Add a dedicated entry for guest_memfd, with the usual suspects as Maintainers, and David Hildenbrand as a Reviewer. - Add Sean as a Reviewer for overall KVM. |
||
|
Sean Christopherson
|
f8479adb28 |
KVM: guest_memfd: Move gmem function declarations to dedicated guest_memfd.h
Extract the gmem function declarations out of kvm_mm.h and into a dedicated header, guest_memfd.h. This will allow creating a MAINTAINERS entry for guest_memfd without having to rely on content pattern matching. Reviewed-by: Ackerley Tng <[email protected]> Acked-by: David Hildenbrand (Arm) <[email protected]> Link: https://patch.msgid.link/[email protected] Signed-off-by: Sean Christopherson <[email protected]> |
||
|
Lorenzo Stoakes
|
93d23dff8e |
mm: use linear_page_[index, delta]() consistently
There are a number of places where we open code what linear_page_index() and linear_page_delta() calculate. Replace this code with the appropriate functions for consistency. No functional change intended. Link: https://lore.kernel.org/[email protected] Signed-off-by: Lorenzo Stoakes <[email protected]> Reviewed-by: Thomas Zimmermann <[email protected]> [DRM] Acked-by: Kai Huang <[email protected]> # for sgx Reviewed-by: Gregory Price <[email protected]> Reviewed-by: Pedro Falcato <[email protected]> # for mm Reviewed-by: Ackerley Tng <[email protected]> [guest_memfd] Reviewed-by: Vlastimil Babka (SUSE) <[email protected]> Cc: David Hildenbrand (Arm) <[email protected]> Cc: Marek Szyprowski <[email protected]> Cc: SJ Park <[email protected]> Cc: Liam R. Howlett (Oracle) <[email protected]> Cc: Zi Yan <[email protected]> Signed-off-by: Andrew Morton <[email protected]> |
||
|
Dmytro Maluka
|
97d65b544f |
KVM: Check for duplicate vcpu_id as early as possible
If userspace tries to create a vCPU with the same vcpu_id as an existing
one, kvm_vm_ioctl_create_vcpu() checks for that and fails with -EEXIST
only after it already created the vCPU via kvm_arch_vcpu_create(). As a
result, even though this newly created vCPU is destroyed in the failure
path, the fact that it is temporarily created with an invalid vcpu_id
and that there are temporarily two vCPUs with the same vcpu_id is a
potential source of subtle issues.
In particular, this prevents fixing an VMX IPIv issue where a stale entry
left in the VM's PI descriptor table after the vCPU is destroyed in the
failure path. The right way to fix that issue is to clear that entry when
destroying the vCPU, however right now that would have a nasty side effect:
since the same entry is used for the other, previously created vCPU with
same vcpu_id, clearing it would mean effectively disabling IPIv for that
existing good vCPU.
So to avoid this and similar problems, check for duplicate vcpu_id as
early in the vCPU creation path as possible, before
kvm_arch_vcpu_create() and even before kvm_arch_vcpu_precreate().
Simply moving the existing kvm_get_vcpu_by_id() check earlier doesn't work,
as kvm->lock is dropped and reacquired, i.e. moving kvm_get_vcpu_by_id()
would introduce a race:
1. vCPU A is being created but not installed in kvm->vcpu_array yet.
2. vCPU B with the same vcpu_id is being created. It passes the
duplicated vcpu_id check, since the check doesn't find vCPU A in
kvm->vcpu_array.
3. vCPU A is installed in kvm->vcpu_array, vCPU creation succeeds.
4. vCPU B with the same vcpu_id is installed in kvm->vcpu_array, vCPU
creation succeeds.
So introduce the bitmap of vcpu_ids used by the VM, in order to safely
check if the given vcpu_id is used and mark is as used before releasing
kvm->lock first time.
Alternatively, KVM could use another Xarray[*] for roughly the same code
complexity, which would minimize KVM's steady state memory footprint at
the cost of higher runtime latency (to allocate and free entries). Given
that the worst case scenario is 256 bytes per-VM (on x86, which allows up
to 16KiB vCPU IDs), go with the slightly simpler approach until there's a
need to save memory.
Suggested-by: Sean Christopherson <[email protected]>
Link: https://lore.kernel.org/kvm/[email protected] [*]
Signed-off-by: Dmytro Maluka <[email protected]>
Reviewed-by: Kai Huang <[email protected]>
Link: https://patch.msgid.link/[email protected]
[sean: massage changelog]
Signed-off-by: Sean Christopherson <[email protected]>
|
||
|
Sean Christopherson
|
e186d4e34a |
KVM: Harden kvm_vcpu_map() against double-mapping and thus leaking references
Now that all on-stack maps use CLASS(kvm_vcpu_map_local), i.e. now that all maps are zero-allocated, explicitly put any existing mappings/references when establishing a new mapping to harden against KVM bugs leaking memory, but yell loudly as the owner of the map is still ultimately responsible for the lifecycle of the mapping. Suggested-by: Yosry Ahmed <[email protected]> Link: https://patch.msgid.link/[email protected] Signed-off-by: Sean Christopherson <[email protected]> |
||
|
Sean Christopherson
|
2abcdf03fd |
KVM: guest_memfd: Make private exactly what can be mapped on page fault
When calling into arch code to make the underlying memory private, i.e. to
assign memory to the VM in SNP's RMP table, assign/convert *exactly* the
range of memory that can be mapped into the guest for the current page
fault, instead of aggressively converting/assigning the entire folio. For
SNP, the mapping size in the stage-2 page tables (Nested Page Tables, NPT)
must be at least the size of the corresponding RMP entry, e.g. assigning a
2MiB mapping in the RMP when it can only be mapped at 4KiB granualarity
will ultimate result in another page fault (#NPF for SNP) to "smash" the
RMP down to the correct mapping size.
Assigning the entire folio was necessary back when guest_memfd tracked
preparedness, which was done on a per-folio basis. At the time, it made
sense to do per-folio tracking/preparation, because tracking per-folio
meant guest_memfd didn't need to add a separate data structure to track
that information, and doing per-folio tracking only works if the entire
folio is prepared (or not).
Now that guest_memfd no longer does preparation tracking (see commit
|
||
|
Sean Christopherson
|
fb50ca77b6 |
KVM: guest_memfd: Explicitly pass number of pages to make_private() hook
Tweak the guest_memfd make_private() hook to explicitly pass the number of
pages to align with the signature of the make_shared() hook, and because
the existing code is outright broken if a guest_memfd folio is comprised of
more than one page (which can't happen, yet). The SNP code *tries* to
create a corresponding huge entry, but if the RMP must use 4KiB entries for
whatever reason, KVM will only convert the first pfn, and not the entire
range of pfns that will be mapped into the guest.
Alternatively, @max_order could simply be repurposed as _the_ @order, but
that will fall apart when in-place conversion comes along, at which point
KVM will need to deal with conversions that aren't bound 1:1 to a folio.
I.e. the number of pages to convert may not be exactly a power-of-2 (and
folios don't strictly guarantee power-of-2 pages anyways).
WARN in the SNP code if the number of pages to prepare is anything other
than '1', i.e. if guest_memfd is trying to prepare/convert more than a
single 4KiB page, as sev_gmem_prepare() doesn't actually handle conversion
greater than order-0 folios.
Opportunistically swap the ordering of @pfn and @gfn params for
kvm_x86_ops.gmem_make_private() to match kvm_arch_gmem_make_private().
Fixes:
|
||
|
Sean Christopherson
|
2131c4f763 |
KVM: guest_memfd: Rename prepare() hook and Kconfig to make_private() / CONVERT
Rework guest_memfd's prepare() hook into a more accurate make_private(), and rework its Kconfig from PREPARE to a more generic CONVERT. This will allow x86 to share (pun intended) a kvm_x86_ops.gmem_make_shared() hook between the "convert to shared" and "reclaim" flows, which are one and the same for SNP. No functional change intended. Reviewed-by: Ackerley Tng <[email protected]> Reviewed-by: Fuad Tabba <[email protected]> Link: https://patch.msgid.link/[email protected] Signed-off-by: Sean Christopherson <[email protected]> |
||
|
Ackerley Tng
|
3df611b5a0 |
KVM: guest_memfd: Only "prepare" folios for private pages
When getting a guest_memfd pfn, prepare the folio, i.e. convert its pages to private, if and only if the page is actually private. The misnamed prepare() hook exists specifically to allow x86's SNP to assign pages to the owning VM in the RMP when mapping private memory into a guest. Guarding the call will allow renaming the prepare() hook to better reflect its role, without creating a semantic mess, and will become a hard requirement once in-place conversion is supported, i.e. when CoCo VMs support SHARED guest_memfd pages. For all intents, no functional change intended (the sole arch hook is a nop for SHARED memory). Suggested-by: Michael Roth <[email protected]> Reviewed-by: Fuad Tabba <[email protected]> [sean: rewrite changelog to fit the context] Signed-off-by: Ackerley Tng <[email protected]> Reviewed-by: Xiaoyao Li <[email protected]> Link: https://patch.msgid.link/[email protected] Signed-off-by: Sean Christopherson <[email protected]> |
||
|
Sean Christopherson
|
19e612cd59 |
KVM: guest_memfd: Add helpers to query SHARED vs. PRIVATE for a given page
Add helpers to check if a given page in a guest_memfd instance is PRIVATE versus SHARED, and use the "is shared" helper instead of an open-coded equivalent in the user pagefault handler. In addition to the immediate usage, providing an "is private" helper will allow cleaning up the so called prepare() code, and eventually will be heavily used once in-place conversion support comes along. No functional change intended. Reviewed-by: Xiaoyao Li <[email protected]> Reviewed-by: Ackerley Tng <[email protected]> Reviewed-by: Fuad Tabba <[email protected]> Link: https://patch.msgid.link/[email protected] Signed-off-by: Sean Christopherson <[email protected]> |
||
|
Sean Christopherson
|
0f68fabe68 |
KVM: guest_memfd: Drop the redundant printk on arch gmem_prepare() failure
Drop guest_memfd's ratelimited printk to log "preparation" failures, as KVM SNP already logs more precise messages in all error paths, and whether or not failure to convert the pfn to private is "unexpected", i.e. warrants logging, is firmly an architecture specific detail. Reviewed-by: Ackerley Tng <[email protected]> Reviewed-by: Xiaoyao Li <[email protected]> Reviewed-by: Fuad Tabba <[email protected]> Link: https://patch.msgid.link/[email protected] Signed-off-by: Sean Christopherson <[email protected]> |
||
|
Sean Christopherson
|
7b8529e70a |
KVM: guest_memfd: Rename invalidate() arch hook to reclaim() and isolate it
Rename guest_memfd's invalidate() hook to reclaim() and isolate it via its own RECLAIM Kconfig, as the hook is called when a folio is freed, which is far too late and lacks sufficient information for KVM to actually invalidate its usage of the memory. E.g. SNP uses the hook to convert memory back to SHARED so that it can be safely accessed by the host, there is no invalidation of guest mappings anywhere. Isolating the hook will also allow pKVM on arm64 to opt-in to reclaim() without also having to differentiate between reclaim and conversions to shared for active VMs. Keep guest_memfd's trampoline, even though it would be trivial to wire up .free_folio() directly to an arch callback, to avoid bleeding guest_memfd internals into arch code (specifically, avoid referencing folios in arch code). Leave the kvm_x86_ops hook as-is for the moment, as "reclaim" on SNP is the same as convert-to-shared, i.e. using a different name for the x86 hook will allow reusing it for in-place conversion. Reviewed-by: Xiaoyao Li <[email protected]> Reviewed-by: Fuad Tabba <[email protected]> Reviewed-by: Ackerley Tng <[email protected]> Link: https://patch.msgid.link/[email protected] Signed-off-by: Sean Christopherson <[email protected]> |
||
|
Sean Christopherson
|
0f0893ac5a |
KVM: guest_memfd: Pass the number of pages instead of the end pfn into .invalidate()
Pass the number of pages to "invalidate", i.e. reclaim, instead of the end
pfn, as a first step towards aligning the function prototypes between the
de facto "to private" and "to shared" arch hooks. Eventually, the goal
is to end up with kvm_gmem_arch_make_{private,shared}(), and in both cases,
providing the number of pages makes the call sites slightly nicer, and also
avoids any confusion over whether the end pfn is inclusive or exclusive.
Opportunistically rename "start" to "pfn", again to align with the expected
signature of make_private() (which needs to pass a starting gfn as well, at
which point the "start" becomes noise).
No functional change intended.
Cc: Fuad Tabba <[email protected]>
Cc: Ackerley Tng <[email protected]>
Reviewed-by: Xiaoyao Li <[email protected]>
Reviewed-by: Ackerley Tng <[email protected]>
Reviewed-by: Fuad Tabba <[email protected]>
Link: https://patch.msgid.link/[email protected]
Signed-off-by: Sean Christopherson <[email protected]>
|
||
|
Paolo Bonzini
|
15f8ba7806 |
Merge tag 'kvmarm-fixes-7.2-2' of git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm into HEAD
KVM/arm64 fixes for 7.2, take #2 - Move locking for kvm_io_bus_get_dev() into the caller, ensuring race-free checks that the returned object is of the correct type - Fix initialisation of the page-table walk level when relaxing permissions - Correctly update the XN attribute when relaxing permissions - Fix the sign extension of loads from emulated MMIO regions - Assorted collection of fixes for pKVM's FFA proxy, together with a couple of FFA driver adjustments |
||
|
Sean Christopherson
|
d1a3c21623 |
KVM: SEV: Forcefully invalidate SNP VMSA if its backing gmem page is zapped
Wire up a gmem_invalidate_range() call for SNP VMs, and use it to force
vCPUs to reload/recheck their guest-provided VMSA if the backing gmem
page is being invalidated, e.g. is being PUNCH_HOLE'd. Use the same core
logic to handle invalidations as VMX does for the APIC-access page, as the
two concepts are nearly identical: shove the physical address of a page
into the vCPU's control structure:
1. Snapshot the invalidation sequence counter
2. Grab the pfn (from guest_memfd in this case)
3. Acquire mmu_lock for read
4. Re-request reload if retry is needed, otherwise commit the change.
Note, the re-request action in #4 is necessary as KVM's retry logic is
fuzzy, i.e. can get false positives. If the guest_memfd page has been
dropped, at some point a subsequent reload will fail to get a PFN from
guest_memfd, and KVM will fail KVM_RUN. If the retry was due to a false
positive, KVM will retry until there are no relevant MMU notifier events
(and will retry in the "outer" loop, i.e. will drop locks and resched as
needed).
Note #2! Take care to invalidate the VMSA when a relevant memslot is
DELETED or MOVED, as invalidations in response to PUNCH_HOLE are predicated
on memslot bindings (KVM doesn't know what GFN range(s) to invalidate
without a binding). And more importantly, the VMSA mapping requires a
memslot, i.e. must be invalidated if its memslots disappears, regardless of
the state of the underlying guest_memfd inode.
Failure to invalidate the vCPU's control.vmsa_pa (which is checked by
pre_sev_run()) can prevent KVM from properly freeing the page as firmware
will reject the RMPUPDATE to reclaim the page with FAIL_INUSE if the vCPU
is actively running, i.e. if VMSA page is in-use. That in turn leads to an
RMP #PF on the next use, as the page will still be assigned to the SNP VM.
SEV-SNP: RMPUPDATE failed for PFN 78d198, pg_level: 1, ret: 3
SEV-SNP: PFN 0x78d198, RMP entry: [0xfff0000000144001 - 0x000000000000000f]
CPU: 3 UID: 0 PID: 31345 Comm: sev_snp_vmsa_pu Tainted: G U O
Tainted: [U]=USER, [O]=OOT_MODULE
Hardware name: Google, Inc. Arcadia_IT_80/Arcadia_IT_80, BIOS 34.86.0-102 01/25/2026
Call Trace:
<TASK>
dump_stack_lvl+0x54/0x70
rmpupdate+0x12c/0x140
rmp_make_shared+0x3b/0x60
sev_gmem_invalidate+0xe0/0x170 [kvm_amd]
delete_from_page_cache_batch+0x1d8/0x220
truncate_inode_pages_range+0x120/0x3d0
kvm_gmem_fallocate+0x19a/0x270 [kvm]
vfs_fallocate+0x1bc/0x1f0
__x64_sys_fallocate+0x48/0x70
do_syscall_64+0x10a/0x480
entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x496c7e
</TASK>
------------[ cut here ]------------
SEV: Failed to update RMP entry for PFN 0x78d198 error -14
WARNING: arch/x86/kvm/svm/sev.c:5160 at sev_gmem_invalidate+0x126/0x170 [kvm_amd], CPU#3: sev_snp_vmsa_pu/31345
CPU: 3 UID: 0 PID: 31345 Comm: sev_snp_vmsa_pu Tainted: G U O
Tainted: [U]=USER, [O]=OOT_MODULE
Hardware name: Google, Inc. Arcadia_IT_80/Arcadia_IT_80, BIOS 34.86.0-102 01/25/2026
RIP: 0010:sev_gmem_invalidate+0x12b/0x170 [kvm_amd]
Call Trace:
<TASK>
delete_from_page_cache_batch+0x1d8/0x220
truncate_inode_pages_range+0x120/0x3d0
kvm_gmem_fallocate+0x19a/0x270 [kvm]
vfs_fallocate+0x1bc/0x1f0
__x64_sys_fallocate+0x48/0x70
do_syscall_64+0x10a/0x480
entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x496c7e
</TASK>
irq event stamp: 20689
hardirqs last enabled at (20699): [<ffffffff8e76092c>] __console_unlock+0x5c/0x60
hardirqs last disabled at (20708): [<ffffffff8e760911>] __console_unlock+0x41/0x60
softirqs last enabled at (20722): [<ffffffff8e6cd74e>] __irq_exit_rcu+0x7e/0x140
softirqs last disabled at (20717): [<ffffffff8e6cd74e>] __irq_exit_rcu+0x7e/0x140
---[ end trace 0000000000000000 ]---
BUG: unable to handle page fault for address: ffff99a64d198000
#PF: supervisor write access in kernel mode
#PF: error_code(0x80000003) - RMP violation
PGD 13eb001067 P4D 13eb001067 PUD 78d1d1063 PMD 1184e0063 PTE 800000078d198163
SEV-SNP: PFN 0x78d198, RMP entry: [0x6030000000144001 - 0x000000000000000f]
Oops: Oops: 0003 [#1] SMP
CPU: 3 UID: 0 PID: 31407 Comm: highlanderd_hea Tainted: G U W O
Tainted: [U]=USER, [W]=WARN, [O]=OOT_MODULE
Hardware name: Google, Inc. Arcadia_IT_80/Arcadia_IT_80, BIOS 34.86.0-102 01/25/2026
RIP: 0010:prep_new_page+0x67/0x220
Call Trace:
<TASK>
get_page_from_freelist+0x1c40/0x1c70
__alloc_frozen_pages_noprof+0xca/0x1f0
alloc_pages_mpol+0x10b/0x1b0
alloc_pages_noprof+0x81/0x90
pte_alloc_one+0x1b/0xd0
do_pte_missing+0xdf/0x1020
handle_mm_fault+0x7c7/0xb20
do_user_addr_fault+0x268/0x6b0
exc_page_fault+0x67/0xa0
asm_exc_page_fault+0x26/0x30
RIP: 0033:0x4a6b1e
</TASK>
gsmi: Log Shutdown Reason 0x03
CR2: ffff99a64d198000
---[ end trace 0000000000000000 ]---
RIP: 0010:prep_new_page+0x67/0x220
Drop the pseudo-TODO comment about needing to pin the page if guest_memfd
every supports migration, as integrating with invalidations events means
KVM will Just Work if/when page migration is ever supported (assuming SNP
hardware supports migrating VMSA pages).
Note #3, invalidate() and invalidate_range() have _completely_ different
semantics; the new invalidate_range() is a true invalidation, whereas the
existing invalidate() is really a "make shared" operation. Ignore the
confusing naming and poor Kconfig bundling for the moment to minimize the
delta for LTS kernels, the mess will be cleaned up shortly.
Reported-by: Hyunwoo Kim <[email protected]>
Closes: https://lore.kernel.org/all/aimMWzAf5b3luM0b@v4bel
Fixes:
|
||
|
leixiang
|
db3a46e200 |
KVM: Remove kvm_debugfs_dir on kvm_init() error paths
kvm_init_debug() runs before several steps that can fail
(kvm_vfio_ops_init(), kvm_gmem_init(), kvm_init_virtualization() and
misc_register()), but none of the corresponding error labels remove the
"kvm" debugfs directory. Any failure after kvm_init_debug() therefore
leaks the directory and its stat files for the lifetime of the boot.
kvm_exit() already calls debugfs_remove_recursive(kvm_debugfs_dir); add
the same at the err_vfio label, whose fall-through covers every path
taken after kvm_init_debug().
Fixes:
|
||
|
leixiang
|
b31e7c24b4 |
KVM: Remove kvm_debugfs_dir on kvm_init() error paths
kvm_init_debug() runs before several steps that can fail
(kvm_vfio_ops_init(), kvm_gmem_init(), kvm_init_virtualization() and
misc_register()), but none of the corresponding error labels remove the
"kvm" debugfs directory. Any failure after kvm_init_debug() therefore
leaks the directory and its stat files for the lifetime of the boot.
kvm_exit() already calls debugfs_remove_recursive(kvm_debugfs_dir); add
the same at the err_vfio label, whose fall-through covers every path
taken after kvm_init_debug().
Fixes:
|
||
|
Sean Christopherson
|
b29125ead0 |
KVM: Initialize a vCPU's index to '-1' while it's being created
Invalidate a vCPU's index immediately after allocating storage for the vCPU so that KVM doesn't incorrectly treat a vCPU that is the process of being created as being vCPU0. This will also allow detecting that a vCPU is in the process of being created and thus otherwise unreachable, which is useful for avoiding false positives in lockdep assertions on vcpu->mutex. Unwind the index back to -1 if inserting the vCPU into the array or adding the vCPU to the fd table fails, so that kvm_arch_vcpu_destroy() sees the vCPU as unreachable, i.e. so that teardown logic doesn't hit false positive lockdep assertions. Opportunistically add a comment to call out that the "real" index needs to be set before making the vCPU visible to other tasks. Note, kvm_wait_for_vcpu_online() naturally does the right thing thanks to vcpu->vcpu_idx and kvm->online_vcpus being signed values. Reviewed-by: Philippe Mathieu-Daudé <[email protected]> Link: https://patch.msgid.link/[email protected] Signed-off-by: Sean Christopherson <[email protected]> |
||
|
Marc Zyngier
|
3a07249981 |
KVM: Move kvm_io_bus_get_dev() locking responsibilities to callers
kvm_io_bus_get_dev() returns a device that is only matched by the
address, and nothing else. This can cause a lifetime issue if
the matched device is not the expected type, as by the time
the caller can introspect the object, it might be gone (the srcu
lock having been dropped).
Given that there is only a single user of this helper, the simplest
option is to move the locking responsibility to the caller, which
can keep the srcu lock held for as long as it wants.
Note that this aligns with other kvm_io_bus*() helpers, which
already require the srcu lock to be held by the callers.
Reported-by: Will Deacon <[email protected]>
Fixes:
|
||
|
Linus Torvalds
|
c75597caad |
Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm
Pull kvm fixes from Paolo Bonzini:
"s390:
- Fix S390_USER_OPEREXEC so it can now be enabled regardless of other
unrelated capabilities
- Fix handling of the _PAGE_UNUSED pte bit that could lead to guest
memory corruption in some scenarios
- A bunch of misc gmap fixes (locking, behaviour under memory
pressure)
- Fix CMMA dirty tracking
x86:
- Tidy up some WARN_ON() and BUG_ON(), replacing them with
WARN_ON_ONCE() or KVM_BUG_ON(). All of these have obviously never
triggered, or somebody would have been annoyed earlier, but still...
- Fix missing interrupt due to stale CR8 intercept
- Add a statistic that can come in handy to debug leaks as well as
the vulnerability to a class of recently-discovered issues
- Do not ask arch/x86/kernel to export
default_cpu_present_to_apicid() just for KVM"
* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (22 commits)
x86/apic: KVM: Use cpu_physical_id() to get APIC ID of running vCPU for AVIC
KVM: x86/mmu: Expose number of shadow MMU shadow pages as a stat
KVM: x86: Unconditionally recompute CR8 intercept on PPR update
KVM: VMX: Grab vmcs12 on CR8 interception update iff vCPU is in guest mode
KVM: x86: WARN (once) if RTC pending EOI tracking goes off the rails
KVM: x86: WARN and fail kvm_set_irq() if a PIC or I/O APIC vector is invalid
KVM: x86: Bug the VM, not the kernel, if the ISR count {under,over}flows
KVM: x86/mmu: Bug the VM, not the host kernel, if KVM write-protects upper SPTEs
KVM: x86: Replace BUG_ON() with WARN_ON_ONCE() on "bad" nested GPA translation
KVM: Replace guest-triggerable BUG_ON() in ioeventfd datamatch with get_unaligned()
KVM: s390: Return failure in case of failure in kvm_s390_set_cmma_bits()
KVM: s390: selftests: Fix cmma selftest
KVM: s390: Fix cmma dirty tracking
KVM: s390: Fix locking in kvm_s390_set_mem_control()
KVM: s390: Fix handle_{sske,pfmf} under memory pressure
KVM: s390: Fix code typo in gmap_protect_asce_top_level()
KVM: s390: Do not set special large pages dirty
KVM: s390: Fix dat_peek_cmma() overflow
s390/mm: Fix handling of _PAGE_UNUSED pte bit
KVM: s390: Fix typo in UCONTROL documentation
...
|
||
|
Sean Christopherson
|
f1edbed787 |
KVM: Replace guest-triggerable BUG_ON() in ioeventfd datamatch with get_unaligned()
Drop a BUG_ON() that has been reachable since it was first added, way back
in 2009, and instead use get_unaligned() to perform potentially-unaligned
accesses.
For a given store, KVM x86's emulator tracks the entire value in the
destination operand, x86_emulate_ctxt.dst. If the destination is memory,
and the target splits multiple pages and/or is emulated MMIO, then KVM
handles each fragment independently. E.g. on a page split starting at page
offset 0xffc, KVM writes 4 bytes to the first page, then the remaining
bytes to the second page, using ctxt->dst as the source for both (with
appropriate offsets).
If the destination splits a page *and* hits emulated MMIO on the second
page, then KVM will complete the write to the first page, then emulate the
MMIO access to the second page. If there is a datamatch-enabled ioeventfd
at offset 0 of the second page, then KVM will process the remainder of the
store as a potential ioeventfd signal.
Putting it all together, if the guest emits a store that splits a page
starting at page offset N, and the second page has a datamatch-enabled
ioeventfd at offset 0, then KVM will check for datamatch using
&dst.valptr[N] as the source. Due to dst (and thus dst.valptr) being
32-byte aligned, if N is not aligned to @len, the BUG_ON() fires.
E.g. with a 16-byte store at page offset 0xffc, to an ioeventfd of len 8,
all initial checks in ioeventfd_in_range() will succeed, and the BUG_ON()
fires due to @val being 4-byte aligned, but not 8-byte aligned.
------------[ cut here ]------------
kernel BUG at arch/x86/kvm/../../../virt/kvm/eventfd.c:783!
Oops: invalid opcode: 0000 [#1] SMP
CPU: 0 UID: 1000 PID: 615 Comm: repro Not tainted 7.1.0-rc2-ff238429d1ea #365 PREEMPT
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
RIP: 0010:ioeventfd_write+0x6c/0x70 [kvm]
Call Trace:
<TASK>
__kvm_io_bus_write+0x85/0xb0 [kvm]
kvm_io_bus_write+0x53/0x80 [kvm]
vcpu_mmio_write+0x66/0xf0 [kvm]
emulator_read_write_onepage+0x12a/0x540 [kvm]
emulator_read_write+0x109/0x2b0 [kvm]
x86_emulate_insn+0x4f8/0xfb0 [kvm]
x86_emulate_instruction+0x181/0x790 [kvm]
kvm_mmu_page_fault+0x313/0x630 [kvm]
vmx_handle_exit+0x18a/0x590 [kvm_intel]
kvm_arch_vcpu_ioctl_run+0xc81/0x1c90 [kvm]
kvm_vcpu_ioctl+0x2d5/0x970 [kvm]
__x64_sys_ioctl+0x8a/0xd0
do_syscall_64+0xb7/0x890
entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x7f19c931a9bf
</TASK>
Modules linked in: kvm_intel kvm irqbypass
---[ end trace 0000000000000000 ]---
In a perfect world, the fix would be to simply delete the BUG_ON(), as KVM
x86 doesn't perform alignment checks on "normal" memory accesses at CPL0.
Sadly, C99 ruins all the fun; while the x86 architecture plays nice,
dereferencing an unaligned pointer directly is undefined behavior in C,
e.g. triggers splats when running with CONFIG_UBSAN_ALIGNMENT=y.
Fixes:
|
||
|
Linus Torvalds
|
c98d767b34 |
Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm
Pull kvm updates from Paolo Bonzini:
"arm64:
This is a bit of an odd merge window on the KVM/arm64 front. There
is absolutely no new feature in the pull request. It is purely
fixes, because it is simply becoming too hard to review new stuff
when so many AI-fuelled fixes hit the list.
- Significant cleanup of the vgic-v5 PPI support which was merged in
7.1. This makes the code more maintainable, and squashes a couple
of bugs in the meantime
- Set of fixes for the handling of the MMU in an NV context,
particularly VNCR-triggered faults. S1POE support is fixed as well
- Large set of pKVM fixes, mostly addressing recurring issues around
hypervisor tracking of donated pages in obscure cases where the
donation could fail and leave things in a bizarre state
- Fixes for the so-called "lazy vgic init", which resulted in
sleeping operations in non-preemptible sections. This turned out to
be far more invasive than initially expected..
- Reduce the overhead of L1/L2 context switch by not touching the FP
registers
- Fix the way non-implemented page sizes are dealt with when a guest
insist on using them for S2 translation
- The usual set of low-impact fixes and cleanups all over the map
Loongarch:
- On a request for lazy FPU load, load all FPU state that the VM
supports instead of enabling only the part (FPU, LSX or LASX) that
caused the FPU load request
- Some enhancements about interrupt injection
- Some bug fixes and other small changes
RISC-V:
- Batch G-stage TLB flushes for GPA range based page table updates
- Convert HGEI line management to fully per-HART
- Fix missing CSR dirty marking when FWFT state updated via ONE_REG
- Fix stale FWFT feature exposure to Guest/VM
- Speed up dirty logging write faults using MMU rwlock and atomic PTE
updates using cmpxchg() for permission-only changes
- Use flexible array for APLIC IRQ state
- Use kvm_slot_dirty_track_enabled() for logging enable check on a
memslot
- Avoid skipping valid pages in kvm_riscv_gstage_wp_range()
- Avoid skipping valid pages in kvm_riscv_gstage_unmap_range()
- Use endian-specific __lelong for NACL shared memory
S390:
- KVM_PRE_FAULT_MEMORY support
- Support for 2G hugepages
- Support for the ASTFLEIE 2 facility
- Support for fast inject using kvm_arch_set_irq_inatomic
- Fix potential leak of uninitialized bytes
- A few more misc gmap fixes
x86:
- Generic support for the more granular permissions allowed by EPT,
namely "read" (which was previously usurping the U bit) and
separate execution bits for kernel and userspace
- Do not assume that all page tables start with U=1/W=1/NX=0 at the
root, as AMD GMET needs to have U=0 at the root
- Introduce common assembly macros for use within Intel and AMD
vendor-specific vmentry code. This touches the SPEC_CTRL handling,
which is now entirely done in assembly for Intel (by reusing the
AMD code that already existed), and register save/restore which
uses some macro magic to compute the offsets in the struct. Both of
these are preparatory changes for upcoming APX support
- Clean up KVM's register tracking and storage, primarily to prepare
for APX support, which expands the maximum number of GPRs from 16
to 32
- Keep a single copy of the PDPTRs rather than two, since
architecturally there is just one
- Handle EXIT_FASTPATH_EXIT_USERSPACE in vendor code to ensure vendor
code gets a chance to handle things like reaping the PML buffer
- Update KVM's view of PV async enabling if and only if the MSR write
fully succeeds
- Fix a variety of issues where the emulator doesn't honor
guest-debug state, and clean up related code along the way
- Synthesize EPT Violation and #NPF "error code" bits when injecting
faults into L1 that didn't originate in hardware (in which case the
VMCS/VMCB doesn't hold relevant information)
- Add support for virtualizing (well, emulating) AMD's flavor of
CPL>0 CPUID faulting
- Clean up the GPR APIs so that KVM's use of "raw" is consistent, and
fix a variety of minor bugs along the way
- Fix an OOB memory access due to not checking the VP ID when
handling a Hyper-V PV TLB flush for L2
- Fix a bug in the mediated PMU's handling of fixed counters that
allowed the guest to bypass the PMU event filter
- Allow userspace to return EAGAIN when handling SNP and TDX
hypercalls, so the KVM can forward a "retry" status code to the
guest, and reserve all unused error codes for future usage
- Overhaul the TDP MMU => S-EPT code to move as much S-EPT specific
logic as possible into the TDX code, and to funnel (almost) all
S-EPT updates into a single chokepoint. The motivation is largely
to prepare for upcoming Dynamic PAMT support, but the cleanups are
nice to have on their own
- Plug a hole in shadow page table handling, where KVM fails to
recursively zap nested EPT/NPT shadow page tables when the nested
hypervisor tears down its own EPT/NPT page tables from the bottom
up
x86 (Intel):
- Support for nested MBEC (Mode-Based Execute Control), see above in
the generic section; also run with MBEC enabled even for non-nested
mode
- Use the kernel's "enum pg_level" in the TDX APIs instead of the
TDX-Module's level definitions (which are 0-based)
- Rework the TDX memory APIs to not require/assume that guest memory
is backed by "struct page" (in prepartion for guest_memfd hugepage
support)
- Fix a largely benign bug where KVM TDX would incorrectly state it
could emulate several x2APIC MSRs
- Use the "safe" WRMSR API when proxying LBR MSR writes as the
to-be-written value is guest controlled and completely unvalidated
x86 (AMD):
- Support for nested GMET (Guest Mode Execution Trap), see above in
the generic section; also run with GMET enabled even for non-nested
mode
- Fixes and minor cleanups to GHCB handling, on top of the earlier
work already merged into 7.1-rc
- Ensure KVM's copy of CR0 and CR3 are up-to-date prior to invoking
fastpath handlers
- Add support for virtualizing gPAT (KVM previously just used L1's
PAT when running L2)
- Fix goofs where KVM mishandles side effects (e.g. single-step and
PMC updates) when emulating VMRUN
- Fix a variety of bugs in AVIC's handling of x2APIC MSR
interception, most notably where KVM didn't disable interception of
IRR, ISR, and TMR regs
- Add support for virtualizing Host-Only/Guest-Only bits in the
mediated PMU
- Don't advertise support for unusable VM types, and account for VM
types that are disabled by firmware, e.g. to mitigate security
vulnerabilities
- Rewrite the SEV {en,de}crypt debug ioctls as they were riddle with
bugs and unnecessarily complicated, and add comprehensive tests
- Clean up and deduplicate the SEV page pinning code
- Fix minor goofs related to writing back CPUID information after
firmware rejects a CPUID page for an SNP vCPU
Generic:
- Rename invalidate_begin() to invalidate_start() throughout KVM to
follow the kernel's nomenclature, e.g. for mmu_notifiers
- Use guard() to cleanup up various KVM+VFIO flows
- Minor cleanups
guest_memfd:
- Return -EEXIST instead of -EINVAL if userspace attempts to bind a
gmem range to multiple memslots, and fix the test that was supposed
to ensure KVM returns -EEXIST
- Treat memslot binding offsets and sizes as unsigned values to fix a
bug where KVM interprets a large "offset + size" as a negative
value and allows a nonsensical offset
- Use the inode number instead of the page offset for the NUMA
interleaving index to fix a bug where the effective index would
jump by two for consecutive pages (the caller also adds in the page
offset)
Selftests:
- Randomize the dirty log test's delay when reaping the bitmap on the
first pass, as always waiting only 1ms hid a KVM RISC-V bug as the
test reaped the bitmap before KVM could build up enough state to
hit the bug
- A pile of one-off fixes and cleanups"
* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (326 commits)
KVM: x86/mmu: Ensure hugepage is in by slot before checking max mapping level
KVM: x86: Fix shadow paging use-after-free due to unexpected role
KVM: s390: Introducing kvm_arch_set_irq_inatomic fast inject
KVM: s390: Enable adapter_indicators_set to use mapped pages
KVM: s390: Add map/unmap ioctl and clean mappings post-guest
riscv: kvm: Use endian-specific __lelong for NACL shared memory
KVM: selftests: access_tracking_perf_test: bump number of NUMA nodes to 32
KVM: s390: vsie: Implement ASTFLEIE facility 2
KVM: s390: vsie: Refactor handle_stfle
s390/sclp: Detect ASTFLEIE 2 facility
KVM: s390: Minor refactor of base/ext facility lists
KVM: x86/mmu: move pdptrs out of the MMU
KVM: x86: check that kvm_handle_invpcid is only invoked with shadow paging
KVM: nSVM: invalidate cached PDPTRs across nested NPT transitions
KVM: nVMX: remove unnecessary code in prepare_vmcs02_rare
KVM: x86: remove nested_mmu from mmu_is_nested()
KVM: arm64: vgic-its: Make ABI commit helpers return void
KVM: s390: Initialize KVM_S390_GET_CMMA_BITS memory
LoongArch: KVM: Add missing slots_lock for device register/unregister
LoongArch: KVM: Validate irqchip index in irqfd routing
...
|
||
|
Linus Torvalds
|
7e0e7bd60d |
Merge tag 'vfs-7.2-rc1.misc' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull misc vfs updates from Christian Brauner:
"Features:
- Reduce pipe->mutex contention by pre-allocating pages outside the
lock in anon_pipe_write().
anon_pipe_write() called alloc_page() once per page while holding
pipe->mutex. The allocation can sleep doing direct reclaim and runs
memcg charging, which extends the critical section and stalls any
concurrent reader on the same mutex. Now up to 8 pages are
pre-allocated before the mutex is taken, leftovers are recycled
into the per-pipe tmp_page[] cache before unlock, and any remainder
is released after unlock, keeping the allocator out of the critical
section on both sides. On a writers x readers sweep with 64KB
writes against a 1 MB pipe throughput improves 6-28% and average
write latency drops 5-22%; under memory pressure - when the cost of
holding the mutex across reclaim is highest - throughput improves
21-48% and latency drops 17-33%. The microbenchmark is added to
selftests.
- uaccess/sockptr: fix the ignored_trailing logic in
copy_struct_to_user() to behave as documented and the usize check
in copy_struct_from_sockptr() for user pointers, and add
copy_struct_{from,to}_bounce_buffer() and copy_struct_to_sockptr()
helpers for upcoming users (IPPROTO_SMBDIRECT, IPPROTO_QUIC).
- bpf: add a sleepable bpf_real_inode() kfunc that resolves the real
inode backing a dentry via d_real_inode(). On overlayfs the inode
attached to the dentry doesn't carry the underlying device
information; this is used by the filesystem restriction BPF program
that was merged into systemd.
- docs: add guidelines for submitting new filesystems, motivated by
the maintenance burden abandoned and untestable filesystems impose
on VFS developers, blocking infrastructure work like folio
conversions and iomap migration.
Fixes:
- libfs: set SB_I_NOEXEC and SB_I_NODEV by default in init_pseudo()
and drop the now-redundant assignments in callers. This began as a
one-line dma-buf fix for a path_noexec() warning; a pseudo
filesystem has no reason not to set SB_I_NOEXEC. All init_pseudo()
callers were audited: the only visible effect is on dma-buf where
SB_I_NOEXEC silences the warning.
- Handle set_blocksize() failures in legacy filesystems (bfs, hpfs,
qnx4, jfs, befs, affs, isofs, minix, ntfs3, omfs). Mounting a
device with a sector size > PAGE_SIZE crashed roughly half of them;
the rest had the same missing error handling pattern. Plus a
follow-up releasing the superblock buffer_head when setting the
minix v3 block size fails.
- mount: honour SB_NOUSER in the new mount API.
- fs/fcntl: fix a SOFTIRQ-unsafe lock order in fasync signaling by
switching the process-group paths of send_sigio() and send_sigurg()
from read_lock(&tasklist_lock) to RCU, matching the single-PID
path.
- vfs: add an FS_USERNS_DELEGATABLE flag and set it for NFS, fixing
delegated NFS mounts (fsopen() in a container with the mount
performed by a privileged daemon) that broke when non-init
s_user_ns was tied to FS_USERNS_MOUNT.
- selftests/namespaces: fix a hang in nsid_test where an unreaped
grandchild kept the TAP pipe write-end open, a waitpid(-1) race in
listns_efault_test, and a false FAIL on kernels without listns()
where the tests should SKIP.
- filelock: fix the break_lease() stub signature for
CONFIG_FILE_LOCKING=n.
- init/initramfs_test: wait for the async initramfs unpacking before
running; the test and do_populate_rootfs() share the parser state.
- fs/coredump: reduce redundant log noise in
validate_coredump_safety().
- iomap: pass the correct length to fserror_report_io() in
__iomap_write_begin().
- backing-file: fix the backing_file_open() kerneldoc.
Cleanups:
- initramfs: refactor the cpio hex header parsing to use hex2bin()
instead of the hand-rolled simple_strntoul() which is reverted, and
extend the initramfs KUnit tests to cover header fields with 0x
prefixes.
- Replace __get_free_pages() and friends with kmalloc()/kzalloc()
across quota, proc, ocfs2/dlm, nilfs2, nfs, nfsd, libfs, jfs, jbd2,
isofs, fuse, select, namespace, configfs, binfmt_misc, bfs, and the
do_mounts init code - part of the larger work of replacing page
allocator calls with kmalloc().
- Use clear_and_wake_up_bit() in unlock_buffer() and
journal_end_buffer_io_sync() instead of open-coding the sequence.
- Drop unused VFS exports: unexport drop_super_exclusive(), remove
start_removing_user_path_at(), and fold __start_removing_path()
into start_removing_path().
- fs/read_write: narrow the __kernel_write() export with
EXPORT_SYMBOL_FOR_MODULES().
- vfs: uapi: retire octal and hex constants in favor of (1 << n) for
the O_ flags. Finding a free bit for a new flag across the
architectures was needlessly hard with the mixed bases.
- dcache: add extra sanity checks of dead dentries in dentry_free()
via a new DENTRY_WARN_ONCE() that also prints d_flags.
- iov_iter: use kmemdup_array() in dup_iter() to harden the
allocation against multiplication overflow.
- fs/pipe: write to ->poll_usage only once.
- vfs: remove an always-taken if-branch in find_next_fd().
- dcache: use kmalloc_flex() for struct external_name in __d_alloc().
- namei: use QSTR() instead of QSTR_INIT() in path_pts().
- sync_file_range: delete dead S_ISLNK code.
- Comment fixes: retire a stale comment in fget_task_next() and fix
assorted spelling mistakes"
* tag 'vfs-7.2-rc1.misc' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: (73 commits)
backing-file: fix backing_file_open() kerneldoc parameter
iomap: pass the correct len to fserror_report_io in __iomap_write_begin
vfs: add FS_USERNS_DELEGATABLE flag and set it for NFS
filelock: fix break_lease() stub signature for CONFIG_FILE_LOCKING=n
vfs: uapi: retire octal and hex numbers in favor of (1 << n) for O_ flags
bpf: add bpf_real_inode() kfunc
fs/read_write: Do not export __kernel_write() to the entire world
libfs: drop redundant SB_I_NOEXEC/SB_I_NODEV in init_pseudo() callers
libfs: set SB_I_NOEXEC and SB_I_NODEV by default in init_pseudo()
mount: honour SB_NOUSER in the new mount API
fs/fcntl: fix SOFTIRQ-unsafe lock order in fasync signaling
selftests/pipe: add pipe_bench microbenchmark
fs/pipe: pre-allocate pages outside pipe->mutex in anon_pipe_write
fs: retire stale comment in fget_task_next()
fs: fix spelling mistakes in comment
bfs: replace get_zeroed_page() with kzalloc()
binfmt_misc: replace __get_free_page() with kmalloc()
configfs: replace __get_free_pages() with kzalloc()
fs/namespace: use __getname() to allocate mntpath buffer
fs/select: replace __get_free_page() with kmalloc()
...
|
||
|
Paolo Bonzini
|
3ef8a08e22 |
Merge tag 'kvm-x86-vfio-7.2' of https://github.com/kvm-x86/linux into HEAD
KVM VFIO changes for 7.2 Use guard() to cleanup up various KVM+VFIO flows. |
||
|
Paolo Bonzini
|
b39c6bd6fd |
Merge tag 'kvm-x86-sev-7.2' of https://github.com/kvm-x86/linux into HEAD
KVM SEV changes for 7.2
- Don't advertise support for unusuable VM types, and account for VM types
that are disabled by firmware, e.g. to mitigate security vulnerabilities.
- Rewrite the SEV {en,de}crypt debug ioctls as they were riddle with bugs and
unnecessarily complicated, and add comprehensive tests.
- Clean up and deduplicate the SEV page pinning code.
- Fix minor goofs related to writing back CPUID information after firmware
rejects a CPUID page for an SNP vCPU.
|
||
|
Paolo Bonzini
|
06ba195f13 |
Merge tag 'kvm-x86-gmem-7.2' of https://github.com/kvm-x86/linux into HEAD
KVM guest_memfd changes for 7.2 - Return -EEXIST instead of -EINVAL if userspace attempts to bind a gmem range to multiple memslots, and fix the test that was supposed to ensure KVM returns -EEXIST. - Treat memslot binding offsets and sizes as unsigned values to fix a bug where KVM interprets a large "offset + size" as a negative value and allows a nonsensical offset. - Use the inode number instead of the page offset for the NUMA interleaving index to fix a bug where the effective index would jump by two for consecutive pages (the caller also adds in the page offset). |
||
|
Paolo Bonzini
|
4d37da77e6 |
Merge tag 'kvm-x86-generic-7.2' of https://github.com/kvm-x86/linux into HEAD
KVM generic changes for 7.2 - Rename invalidate_begin() to invalidate_start() throughout KVM to follow the kernel's nomenclature, e.g. for mmu_notifiers. - Minor cleanups. |
||
|
Michael S. Tsirkin
|
48dbe47321 |
KVM: guest_memfd: fix NUMA interleave index double-counting
kvm_gmem_get_policy() sets the interleave index (the output param that's typically named "ilx") to the full page offset (vm_pgoff + vma offset). But get_vma_policy() adds the page offset on top of the interleave index, and so the offset is counted twice. This causes NUMA interleaving to skip nodes: for order-0 pages the effective index jumps by 2 for each consecutive page. The vm_op.get_policy() implementation should return only a per-file bias in the interleave index (like shmem_get_policy does with inode->i_ino), letting get_vma_policy() add the page-offset component. Fix by setting the output interleave index to the inode number (a la shmem) instead of the full page offset, as the index is intended to be a constant, semi-random value for a given file, e.g. so that interleaving doesn't start at the same node for every file, and so that allocations are round-robined across nodes based on the page offset (the selected node would bounce/skip around if the index isn't constant). Found by Sashiko (sashiko.dev) AI code review. Fixes: |
||
|
John Hubbard
|
be5748d2ae |
libfs: drop redundant SB_I_NOEXEC/SB_I_NODEV in init_pseudo() callers
init_pseudo() now sets SB_I_NOEXEC and SB_I_NODEV by default, so the per-caller assignments are redundant. Drop them. Signed-off-by: John Hubbard <[email protected]> Link: https://patch.msgid.link/[email protected] Signed-off-by: Christian Brauner (Amutable) <[email protected]> |
||
|
Sean Christopherson
|
8618004d3e |
KVM: Don't WARN if memory is dirtied without a vCPU when the VM is dying
When marking a page dirty, complain about not having a running/loaded vCPU if and only if the VM is still alive, i.e. its refcount is non-zero. This will allow fixing a memory leak for x86 SEV-ES guests without hitting what is effectively a false positive on the WARN. For some SEV-ES VM-Exits, KVM keeps a writable mapping of a guest page across an exit to userspace, and typically unmaps the page on the next KVM_RUN. But if userspace never calls KVM_RUN after such an exit, then KVM needs to unmap the page when the vCPU is destroyed, which in turn triggers the WARN about not having a running vCPU. Alternatively, SEV-ES could temporarily load the vCPU to suppress the WARN, as is done in nested_vmx_free_vcpu() (but for completely unrelated reasons; suppressing WARN from nested_put_vmcs12_pages() is pure happenstance). But loading a vCPU during destruction is gross (ideally nVMX code would be cleaned up), risks complicating the SEV-ES code (KVM would need to ensure the temporarily load()+put() only runs when the vCPU isn't already loaded), and is ultimately pointless. The motivation for the WARN is to guard against KVM dirtying guest memory without pushing the corresponding GFN to the active vCPU's dirty ring, e.g. to ensure userspace doesn't miss a dirty page. But for the VM's refcount to reach zero, there can't be _any_ userspace mappings to the dirty ring, as mapping the dirty ring requires doing mmap() on the vCPU FD. I.e. if userspace had a valid mapping for the dirty ring, then the vCPU file and thus the owning VM would still be alive. And so since userspace can't possibly reach the dirty ring, whether or not KVM technically "misses" a push to the dirty ring is irrelevant. Reported-by: Michael Roth <[email protected]> Cc: [email protected] Reviewed-by: Michael Roth <[email protected]> Signed-off-by: Sean Christopherson <[email protected]> Message-ID: <[email protected]> Signed-off-by: Paolo Bonzini <[email protected]> Message-ID: <[email protected]> Signed-off-by: Paolo Bonzini <[email protected]> |
||
|
Sean Christopherson
|
eba85fee7f |
KVM: guest_memfd: Treat memslot binding offset+size as unsigned values
When binding a memslot to a guest_memfd file, treat the offset and size as
unsigned values to fix a bug where the sum of the two can result in a false
negative when checking for overflow against the size of the file. Passing
unsigned values also avoids relying on somewhat obscure checks in other
flows for safety, and tracks the offset and size as they are intended to be
tracked, as unsigned values.
On 64-bit kernels, the number of pages a memslot contains and thus the size
(and offset) of its guest_memfd binding are unsigned 64-bit values. Taking
the offset+size as an loff_t instead of a uoff_t inadvertently converts
the unsigned value to a signed value if the offset and/or size is massive.
Locally storing the offset and size as signed values is benign in and of
itself (though even that is *extremely* difficult to discern), but
operating on their sum is not.
For the offset, KVM explicitly checks against a negative value, which might
seem like a bug as KVM could incorrectly reject a legitimate binding, but
that's not actually the case as KVM_CREATE_GUEST_MEMFD takes a signed value
for its size, i.e. a would-be-negative offset is also greater than the
maximum possible size of any guest_memfd file.
Regarding the size, while KVM lacks an explicit check for a negative value,
i.e. seemingly has a flawed overflow check, KVM restricts the number of
pages in a single memslot to the largest positive signed 32-bit value:
if (id < KVM_USER_MEM_SLOTS &&
(mem->memory_size >> PAGE_SHIFT) > KVM_MEM_MAX_NR_PAGES)
return -EINVAL;
and so that maximum "size" will ever be is 0x7fffffff000.
The sum of the two is, however, problematic. While the size is restricted
by KVM's memslot logic, the offset is not, i.e. the offset is completely
unchecked until the "offset + size > i_size_read(inode)" check. If the
offset is the (nearly) largest possible _positive_ value, then adding size
to the offset can result in a signed, negative 64-bit value. When compared
against the size of the file (guaranteed to be positive), the negative sum
is always smaller, and KVM incorrectly allows the absurd offset.
Opportunistically add missing includes in kvm_mm.h (instead of relying on
its parents).
Fixes:
|
||
|
Zongyao Chen
|
7ea987a905 |
KVM: guest_memfd: Return -EEXIST for overlapping bindings
KVM_SET_USER_MEMORY_REGION2 rejects guest_memfd ranges that overlap an
existing binding, but kvm_gmem_bind() currently reports the failure through
its generic -EINVAL path. That makes binding conflicts indistinguishable
from malformed guest_memfd parameters.
Return -EEXIST when the target guest_memfd range is already bound, matching
the errno used for overlapping GPA memslots and making the two types of
range conflicts report the same class of error to userspace.
Note, returning -EINVAL was definitely not intentional, as guest_memfd
support was accompanied by a selftest to verify that attempting to create
overlapping bindings fails with -EEXIST. Except the selftest was also
flawed in that it unintentionally overlapped memslot GPAs, and so failed
on KVM's common memslot checks before reaching guest_memfd.
Fixes:
|
||
|
Sean Christopherson
|
f13e900599 |
KVM: SEV: Pin source page for write when adding CPUID data for SNP guest
When populating a guest_memfd instance with the initial CPUID data for an
SNP guest, acquire a writable pin on the source page as KVM will write back
the "correct" CPUID information if the userspace provided data is rejected
by trusted firmware. Because KVM writes to the source page using a kernel
mapping, pinning for read could result in KVM clobbering read-only memory.
Note, well-behaved VMMs are unlikely to be affected, as CPUID information
is almost always dynamically generated by userspace, i.e. it's unlikely for
the CPUID information to be backed by a read-only mapping.
Fixes:
|
||
|
Carlos López
|
156615264f |
KVM: VFIO: update coherency only if file was deleted
When servicing a KVM_DEV_VFIO_FILE_DEL request, if a file is removed from kv->file_list, kv->noncoherent needs to be updated, in case we can revert to using coherent DMA. However, if we found no candidate to remove, there is no need to re-scan the list, so do it only if a matching file was found. To simplify the control flow, use a mutex guard so that we can return early from within the search loop if the maching file is found. Signed-off-by: Carlos López <[email protected]> Reviewed-by: Alex Williamson <[email protected]> Link: https://patch.msgid.link/[email protected] Signed-off-by: Sean Christopherson <[email protected]> |
||
|
Carlos López
|
3b20a19c59 |
KVM: VFIO: deduplicate file release logic
There are two callsites which destroy files in kv->file_list: the function servicing KVM_DEV_VFIO_FILE_DEL, and the relase of the whole KVM VFIO device. The process involves several steps, so move all those into a single function, removing duplicate code. Signed-off-by: Carlos López <[email protected]> Reviewed-by: Alex Williamson <[email protected]> Link: https://patch.msgid.link/[email protected] Signed-off-by: Sean Christopherson <[email protected]> |
||
|
Carlos López
|
40f0f4bb60 |
KVM: VFIO: use mutex guard in kvm_vfio_file_set_spapr_tce()
Use a mutex guard to hold a lock for the entirety of the function, which
removes the need for a goto (whose label even has a misleading name
since
|
||
|
Carlos López
|
9a7ff26392 |
KVM: VFIO: clean up control flow in kvm_vfio_file_add()
The struct file that this function fgets() is always passed to fput() before returning, so use automatic cleanup via __free() to avoid several jumps to the end of the function. Similarly, use a mutex guard to completely remove the need to use gotos. Signed-off-by: Carlos López <[email protected]> Reviewed-by: Alex Williamson <[email protected]> Link: https://patch.msgid.link/[email protected] Signed-off-by: Sean Christopherson <[email protected]> |
||
|
Takahiro Itazuri
|
190cc5370a |
KVM: Rename invalidate_begin to invalidate_start for consistency
Rename kvm_mmu_invalidate_begin() to kvm_mmu_invalidate_start() to align with mmu_notifier_ops.invalidate_range_start(), which is the callback that ultimately drives KVM's MMU invalidation. While the naming within KVM itself is a close split between "_begin" and "_start": $ git grep -E "invalidate(_range)?_begin" **/kvm* | wc -l 12 $ git grep -E "invalidate(_range)?_start" **/kvm* | wc -l 21 All two of the begin() uses are in KVM: $ git grep -E "invalidate(_range)?_begin" * | wc -l 14 And those two holdouts are bugs in invalidate_range_start()'s comment, i.e. will also be fixed sooner or later[*]. On the other hand, use of _start() is pervasive throughout the kernel: $ git grep -E "invalidate(_range)?_start" * | wc -l 117 Even if that weren't the case, conforming to the mmu_notifier_ops naming is the right call since invalidate_range_start() is the external API that KVM hooks into. No functional change intended. Link: https://lore.kernel.org/all/[email protected] [*] Signed-off-by: Takahiro Itazuri <[email protected]> Link: https://patch.msgid.link/[email protected] [sean: massage changelog to provide more (accurate) numbers] Signed-off-by: Sean Christopherson <[email protected]> |