Skip to content

Memory management: per-process VA range, brk frontier tracking, noreserve/LOW_2G support#746

Open
wdcui wants to merge 1 commit intowdcui/stacked/pr5-fs-net-pipesfrom
wdcui/stacked/pr6-mm-address-space
Open

Memory management: per-process VA range, brk frontier tracking, noreserve/LOW_2G support#746
wdcui wants to merge 1 commit intowdcui/stacked/pr5-fs-net-pipesfrom
wdcui/stacked/pr6-mm-address-space

Conversation

@wdcui
Copy link
Copy Markdown
Member

@wdcui wdcui commented Apr 4, 2026

Summary

Extends the memory management subsystem to support per-process virtual address ranges and adds several new capabilities needed by the host process sandbox.

Stack: PR 6 of N — depends on #745

Changes

Per-process VA range (Vmem, PageManager)

  • Vmem::new() and PageManager::new() accept a Range<usize> instead of always using the platform-wide TASK_ADDR_MIN..TASK_ADDR_MAX
  • All address validation (insert_mapping, get_unmmaped_area, handle_page_fault) uses the configured range instead of platform constants
  • Reserved ranges from the platform are clamped to the configured range
  • Cached addr_min/addr_max on PageManager for lock-free bounds checks

Heap (brk) improvements

  • brk_base: floor that prevents shrinking below the initial program break
  • brk_frontier: page-aligned high-water mark that tracks the actual backed extent, replacing the old brk.next_multiple_of(PAGE_SIZE) computation
  • ensure_brk_past(): advances the floor/brk/frontier past non-heap mappings (e.g., trampolines)
  • current_brk() / current_brk_frontier(): read-only accessors

New CreatePagesFlags

  • FD_WRITABLE (1<<7): shared file-backed mapping gets VM_MAYWRITE when the fd is writable
  • NORESERVE (1<<8): sparse reservation without swap/commit upfront
  • LOW_2G (1<<9): constrain mapping below the MAP_32BIT 2 GiB ceiling
  • Backing type widened from u8 to u16

Allocation robustness

  • create_mapping retry loop: non-fixed allocations that fail because the host rejected the address fall back to the next candidate VA instead of returning ENOMEM immediately
  • insert_mapping validates the platform-returned address is within the managed range; frees stray allocations
  • get_unmmaped_area supports require_low_2g and max_start narrowing for the retry loop
  • VmemResizeError::OutOfRange triggers move_mappings fallback in remap_pages

Miscellaneous

  • VmArea.noreserve field, propagated through all split/merge/copy paths
  • may_flags_for_mapping considers fd_writable for VM_MAYWRITE
  • handle_page_fault returns AllocationFailed instead of panicking
  • VmemPageFaultHandler::HANDLE_USER_PAGE_FAULTS const (default false)

Files changed (8 files, +415/-96)

File Change
litebox/src/mm/linux.rs Core Vmem changes: VA range, VmArea noreserve, CreatePagesFlags, retry loop, get_unmmaped_area
litebox/src/mm/mod.rs PageManager: VA range, brk_base/frontier, handle_page_fault, remap OutOfRange
litebox/src/mm/tests.rs Updated Vmem::new and insert_mapping call signatures
litebox_shim_linux/src/lib.rs PageManager::new caller with full platform range
litebox_shim_optee/src/lib.rs PageManager::new caller with full platform range
litebox_shim_linux/src/syscalls/mm.rs register_existing_mapping fd_writable param
litebox_platform_linux_kernel/src/mm/tests.rs Fully-qualified PageManager::new range param
litebox_platform_lvbs/src/mm/tests.rs Fully-qualified PageManager::new range param

Testing

  • cargo build — clean
  • cargo clippy --all-targets --all-features — clean
  • cargo fmt --all -- --check — clean
  • cargo nextest run --profile ci -E 'not test(nine_p)' — 208 passed, 15 failed (pre-existing TUN/EPERM), 34 skipped

@wdcui wdcui force-pushed the wdcui/stacked/pr6-mm-address-space branch from 18a148e to 121fe60 Compare April 4, 2026 00:15
@wdcui wdcui force-pushed the wdcui/stacked/pr5-fs-net-pipes branch from 6448a31 to 046443d Compare April 4, 2026 00:18
@wdcui wdcui force-pushed the wdcui/stacked/pr6-mm-address-space branch 2 times, most recently from 541eb61 to 4722328 Compare April 4, 2026 00:24
…erve/LOW_2G support

- Vmem and PageManager accept a configurable VA range instead of always
  using the platform-wide TASK_ADDR_MIN..TASK_ADDR_MAX, enabling
  per-process address space partitioning.
- Track brk_base and brk_frontier for correct heap management: brk_base
  prevents shrinking below the initial program break, brk_frontier
  tracks the actual page-aligned heap high-water mark.
- Add CreatePagesFlags: FD_WRITABLE (shared file-backed VM_MAYWRITE),
  NORESERVE (sparse reservations), LOW_2G (MAP_32BIT ceiling).
- VmArea gains noreserve field, propagated through all split/merge paths.
- create_mapping uses a retry loop for non-fixed allocations when the
  host rejects an address inside the guest partition.
- get_unmmaped_area supports require_low_2g and max_start narrowing.
- handle_page_fault returns AllocationFailed instead of panicking.
- VmemResizeError::OutOfRange triggers move_mappings fallback in remap.
@wdcui wdcui force-pushed the wdcui/stacked/pr6-mm-address-space branch from 4722328 to 2a84033 Compare April 4, 2026 03:37
@wdcui wdcui marked this pull request as ready for review April 4, 2026 03:43
@wdcui wdcui requested review from CvvT and jaybosamiya-ms April 4, 2026 03:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant