VBoxManage crashes with heap corruption using `guestcontrol run`

[AaronC81]

I recently upgraded VirtualBox and am encountering an issue with the guestcontrol run command of VBoxManage.

Most of the time, the command does not appear to run the process on the guest. (Maybe it is running but not showing me the output, I am not sure.) Instead, VBoxManage fails with exit code -1073740940, which is STATUS_HEAP_CORRUPTION.

> VBoxManage.exe guestcontrol %VM% run --username vboxuser --password changeme -- "C:\Windows\System32\cmd.exe" /c "echo hello"
hello

> echo %errorlevel%
-1073740940

A small amount of the time, the process executes successfully with the expected errorlevel. It seems to be random which occurs, as sometimes the same command will pass or fail between successive attempts.

Debugging

If I use Windows SDK gflags to enable page heap monitoring, and attach WinDbg, I get a heap verifier stop:

===========================================================
VERIFIER STOP 000000000000000F: pid 0x4494: corrupted suffix pattern 

	000002A4F7691000 : Heap handle
	000002A481D32FE0 : Heap block
	0000000000000018 : Block size
	000002A481D32FF8 : corruption address
===========================================================
This verifier stop is not continuable. Process will be terminated 
when you use the `go' debugger command.
===========================================================

The last VBoxManage frame before the stop was VBoxManage + 0xbd04, which called OLEAUT32!SafeArrayDestroy.

I am happy to debug this further if a debug development build, or symbols for the distributed VirtualBox, are available! This is VBoxManage 7.1.0r164728, SHA256 digest of VBoxManage.exe is 5FCFE99C9D231EAA3CC06CBA7AF51E440C065C0F37B08A4D10C69DD978EC284C.

(I tried building from source, but there are some version conflicts with tooling I need for work, so this is a bit tricky for me unfortunately.)

Environment

VirtualBox: I have been able to reproducible this on both VirtualBox 7.0.18 and VirtualBox 7.1.0. The issue is not present on VirtualBox 6.1.28.

Hosts: I have observed this on both of the hosts I have tested, one running Windows 10 and the other Windows 11.

Guests: I have observed this on existing Windows 10 and 11 guests which were originally created in an older VirtualBox version (5.2.16). I also tried creating a brand-new Windows 11 guest on 7.0.18, and the same issue occurred there.

Logs

I have attached the logs from a few attempts at running commands on my newer Windows 11 VM, most of which were unsuccessful.

[pentagonik]

Thanks for the report. Did 7.0.16 work for you in that regard?

[pentagonik]

Can you please try running the same "run" command using VBoxManage and adding "-vvvv" to it? This increases the verbosity and will tell us a little bit more where it appears to crash.

[AaronC81]

(Sorry for the delay, I thought I'd get an email if there were any replies!)

Replying to pentagonik:

Can you please try running the same "run" command using VBoxManage and adding "-vvvv" to it? This increases the verbosity and will tell us a little bit more where it appears to crash.

I tried this and re-ran many times:

VBoxManage.exe guestcontrol %VM% run -vvvv --username vboxuser --password changeme
-- "C:\Windows\System32\cmd.exe" /c "echo hello"

I got this output most of the time - usually exiting with the heap corruption exit code, but sometimes exiting with success:

Executing:
  Image : C:\Windows\System32\cmd.exe
  arg[0]: C:\Windows\System32\cmd.exe
  arg[1]: /c
  arg[2]: echo hello
Creating guest session as user 'vboxuser'...
Waiting for guest session to start...
Successfully started guest session (ID 2)
Starting guest process ...
Process 'C:\Windows\System32\cmd.exe' (PID 6648) started
Wait result is 'started' (1)
Wait result is 'waiting flag not supported' (9)
Wait result is 'waiting flag not supported' (9)
Wait result is 'waiting flag not supported' (9)
Wait result is 'waiting flag not supported' (9)
Wait result is 'waiting flag not supported' (9)
hello
Wait result is 'waiting flag not supported' (9)
Wait result is 'terminated' (2)
Process terminated
Exit code=0 (Status=500 [successfully terminated])
Closing guest session ...

(The number of "Wait result is 'waiting flag not supported'" changes each time, but I can't see an obvious correlation between this and the exit code.)

So it looks like it's executing to completion in most cases before erroring out.

However, occasionally I saw this output instead:

Executing:
  Image : C:\Windows\System32\cmd.exe
  arg[

Where I suppose it's failing much earlier.

---

Did 7.0.16 work for you in that regard?

I haven't tested on this version - my original choice of 7.0.18 was simply what happened to be the latest during the first attempt at updating VirtualBox.

I can test on this version and/or try to "bisect" the problematic version when I get some time :)

[AaronC81]

Replying to pentagonik:

... tell us a little bit more where it appears to crash.

This issue came up for me again, so I spent some time investigating further. Through some debugging of the release VBoxManage binary, and compiling bits of the VirtualBox source in isolation, I've been able to track down exactly where this is crashing.

---

Summary: There appears to be a logic error in the Windows COM implementation of com::SafeArray::push_front, causing it to index one off the end of the allocated array.

The guestcontrol CLI uses this method to collect arguments for run, hence the problem in this ticket.

Source from include/VBox/com/array.h annotated with the issue. I've removed the ifdef-ed out XPCOM bits, to focus on the implementation used on Windows.

// Imagine we've got an empty `SafeArray`, and we call this to insert an element.
bool push_front(const T &aElement)
{
    // On Windows, size and capacity are always the same.
    // So, after this, `size() == 1`.
    if (!ensureCapacity(size() + 1))
        return false;

    // This loop shifts all the elements to the right, to make room for a new one
    // at the beginning. In our case, the loop executes once, with i == 1
    //
    // (This loop is where the issue lies! We have no elements to shift.)
    for (size_t i = size(); i > 0; --i)
    {
        // Copies element at index 0, to element at index 1.
        // But there is no element 1!
        SafeArray::Copy(m.raw[i - 1], m.raw[i]);
    }

    // Insert new item
    SafeArray::Copy(aElement, m.raw[0]);

    return true;
}

The logic for shifting elements thinks the array is one element larger than its actual size. This means it accesses memory off the end of the array every time it is called.

To resolve this, on Windows, the shifting loop needs to start with size() - 1. (On XPCOM platforms, size() is still correct.)

---

VirtualBox 6.1.28 - which does not exhibit this crash - uses push_back to collect arguments, and push_front doesn't yet exist.

I patched my VBoxManage 7.1.0 binary to implement an equivalent change, and it resolves the crash. So I am quite confident this is the specific problem I've been encountering.

---

I unfortunately haven't been able to set up a functional VirtualBox build environment, so I cannot properly test or contribute this change.

If any maintainer is willing to include this - which I would greatly appreciate - I can be responsive to testing a Development build on my environment, to make sure it's fixed the issue.

[pentagonik]

Very interesting find, thank you very much for sharing this! I'll give this a go and keep you posted.

[pentagonik]

Reproduced and fixed the crash. The fix will be available in the next upcoming test build / maintenance release.

Again, thank you for this very valuable contribution!

[AaronC81]

Replying to pentagonik:

Reproduced and fixed the crash. The fix will be available in the next upcoming test build / maintenance release.

Again, thank you for this very valuable contribution!

Amazing - thank you very much for including this fix! I'll keep an eye out for a new build, and test on my end when there's something.

[AaronC81]

Just to close this out - I tested development builds and can confirm that the issue was resolved!

VirtualBox 7.1.10 is the first release version which includes this fix.

[pentagonik]

Thanks for verifying! Closing.