Apple Silicon and Virtual Machines: Beating the 2 VM Limit (2023)

• macOS Internals Deep Dive
• Building a Development Kernel Collection
• Configuring our Mac to boot the Development Kernel Collection
• Putting our machine to work!
• When did Apple grace us with this feature?
• Undoing our work for OS updates
• Closing Thoughts

For those who’ve been wondering what I’ve been doing for these past few months outside of OpenCore Legacy Patcher, I got the amazing opportunity to work as a Mac Admin Intern at a local consultant company.

One of the areas I’ve been working quite a bit with is macOS Virtual Machines, specifically Apple Silicon Virtual Machines based on Apple’s Virtualization framework. However, not long after doing a lot of development and testing using Apple’s VM stack (through the amazing project, UTM), I found a very frustrating limitation: Apple Silicon hosts can only have a maximum of 2 macOS guest VMs active at once.

This is most commonly seen with this error, generated by Virtualization.framework:

The number of virtual machines exceeds the limit. The maximum supported number of active virtual machines has been reached.

The main reason for this error comes from macOS’ SLA, section 2.B.iii:

(iii) to install, use and run up to two (2) additional copies or instances of the Apple Software, or any prior macOS or OS X operating system software or subsequent release of the Apple Software, within virtual operating system environments on each Apple-branded computer you own or control that is already running the Apple Software, for purposes of: (a) software development; (b) testing during software development; (c) using macOS Server; or (d) personal, non-commercial use.

While I cannot officially virtualize more than 2 copies of macOS on a single machine at once for work, I was still interested in figuring out where in macOS Apple embeds these checks and whether hobbyists and researchers could enable support for more than 2 active macOS VMs at once.

macOS Internals Deep Dive

To start, I initially thought this limitation was userspace based and as such would be embedded somewhere within /System/Library/Frameworks/Virtualization.framework. Due to macOS Big Sur’s dyld merger of frameworks, we’ll need to either extract the framework manually or use tooling such as Hopper Disassembler to load specific binaries embedded in the dyld shared cache.

With this, I was able to examine the framework more closely. However, after many hours of research, I was unable to find where Apple imposed the VM limit. At best, I could only determine that the error message was generated from the framework but nowhere in userspace itself does Apple define a hardcoded 2 VM limit…

After a tip from jevinskie on the Hack Different Discord server, I learned that Apple’s guest limitation is implemented somewhere within the closed-source part of XNU (the macOS kernel). While I didn’t have any strings to go off of, I did know that the Intel Kernel won’t have the same code. So with a not-so-quick comparison between the functions and strings of the Intel and Apple Silicon kernels, I found that the main VM stack is under hv_vm_*.

Throwing the development kernel for macOS Sonoma Beta 4 (23A5301h) in IDA, I found the init code for the VM stack: hv_init():

• To get a development kernel, you’ll need to download your OS’ Kernel Debug Kit off the developer portal

Here we see how Apple handles the VM limitation: Using the int hv_apple_isa_vm_quota variable, the kernel decrements/increments the variable as new Virtual Machines as started/stopped.

And something else is interesting, 2 new boot-args: hypervisor= and hv_apple_isa_vm_quota=. The former is a simple gate check for the latter, which is far more interesting: hv_apple_isa_vm_quota= can override the VM limitation in the kernel!

However, after some more research, I found that this logic is not the same in the release kernels. Instead, Apple swapped the hypervisor boot-arg with an AppleInternal check through System Integrity Protection.

Here we have 2 options:

• Boot Apple’s Development Kernel
• Modify the release kernel to strip the AppleInternal check

To save the bit of sanity I have left, we’ll go with the first option. So now my next challenge: Booting a development kernel on my MacBook Pro.

Building a Development Kernel Collection

To build a development kernel collection, we’ll need to fetch the appropriate Kernel Debug Kit from Apple’s Developer Site. Note that KDKs must match the host, otherwise, issues can occur both during kernel and kext linkage as well as during boot.

Once you have the KDK Disk Image downloaded and installed the embedded package, next check the type of kernel your Mac uses:

uname -v | awk -F '/' '{print $NF}'| awk -F '_' '{print $NF}'

On an M2 Pro MacBook Pro (Mac14,9), this will return T6020. Now we can start building our kernel using kmutil create with the appropriate arch, kernel path, and repositories.

Configuring our Mac to boot the Development Kernel Collection

Finally shutdown your Mac, and boot into recovery. Here we’ll set some policies for our machine:

• Disable System Integrity Protection
• Allow custom boot args to be passed
• Configure our Mac to boot our custom Kernel Collection
• Set our boot-args: kcsuffix=development, hypervisor=0x1, hv_apple_isa_vm_quota=0xFF.

Once rebooted, you can verify this applied in Terminal using sysctl kern.osbuildconfig and nvram boot-args.

Putting our machine to work!

Now that everything is prepared, you’ll now want to grab any virtualization solution utilizing Virtualization.framework to run your multiple macOS instances.