We are currently utilizing VZ with Lima (details: Lima VM and VZ) for our development environment. However, we're encountering a critical issue with the com.apple.Virtualization.VirtualMachine process leading to open file handle exhaustion.
When mounting our programming languages dependency cache folder (Which can have a lot of files) into the VZ VM, we encounter an operating system error related to open file limits:
/gomodcache/github.com/go-git/go-git/v5@v5.4.2/plumbing/object/patch.go:14:2: open /gomodcache/github.com/go-git/go-git/v5@v5.4.2/plumbing/format/diff/unified_encoder.go: too many open files in system
Further investigation revealed an abnormally high number of open files associated with the com.apple.Virtualization.VirtualMachine process. A significant portion of these files are not actively used but remain open.
Example Case:
A file (/Users/rcurrah/test.txt) created on the Mac host and listed (ls) in the VM remains open even 20 minutes later, as evidenced by the following command output:
❯ lsof | grep 11208 | grep test.txt
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
com.apple 11208 rcurrah 4823r REG 1,13 0 46200882 /Users/rcurrah/test.txt
Steps to Reproduce the Issue:
To reproduce the file handle exhaustion follow the below steps. This process will create a large number of files on the Mac host, listing them on the VZ VM, and then verifying their open status using lsof.
Setup the VZ Environment with Sharing:
Create a VZ VM with your home directory shared to the VM.
Create a Test Directory on the Mac Host:
Create a new directory on your Mac host, e.g., mkdir ~/test-file-exhaustion.
Generate a Large Number of Files:
Navigate to the created directory: cd ~/test-file-exhaustion.
Use a loop to create a large number of files, e.g., for i in {1..10000}; do touch "file_${i}.txt"; done. This will create 10,000 files named file_1.txt, file_2.txt, etc.
List Files in the VM:
Access the VZ VM shell.
Navigate to the mounted directory and list the files using the ls command, e.g., ls /path/to/mounted/test-file-exhaustion.
Check Open Files on Mac Host:
Exit the VM and return to your Mac host terminal.
Use the lsof command to check for open files related to the com.apple.Virtualization.VirtualMachine process: lsof | grep "$(pgrep com.apple.Virtualization.VirtualMachine)" | grep 'test-file-exhaustion' | wc -l.
Document the Output:
Record the output of the lsof command. Note the number of open files.
Verify File Closure (or Lack Thereof):
After a certain period, e.g., 20 minutes, repeat the lsof command to see if the files are still open, indicating that they haven’t been closed properly by the process.
Given these observations, we have a couple of questions:
Is this behavior of com.apple.Virtualization.VirtualMachine retaining open file handles a known issue or a bug?
Should VZ be managing the closure of these file handles more efficiently, especially when they are no longer in use?
This issue is impacting our development workflow significantly. Any guidance or insights on resolving this would be highly appreciated.
Thank you for your attention to this matter.
Best regards,
Ryan