Getting function addresses from a DllMain hook

Question

Creating DirectX objects directly from DllMain() fails. I have tried starting a new thread, create the objects there and them get vtable/addresses of the functions I'm about to hook.

I'm using Microsoft Detours, DetourTransactionCommit() cannot be called from another thread than DllMain thread.

I have also tried waiting for the thread to finish collecting the addresses, like this:

BOOL WINAPI DllMain(HINSTANCE hinst, DWORD dwReason, LPVOID reserved)
{
    switch (dwReason)
    {
        case DLL_PROCESS_ATTACH:
        {          
            std::thread(InitializeHooks).detach();            
            while (!_done)
            {
                std::this_thread::sleep_for(std::chrono::milliseconds(100));
            }
            // ...
            break;
        }
    }
    
    return TRUE;
}

But doing this, it gets stuck whenever it tries to call any DirectX function.

I'm getting the DirectX function addresses like this:

void InitializeHooks()
{
    if (HMODULE d3d9Module = GetModuleHandle("d3d9.dll"))
    {
        typedef IDirect3D9* (WINAPI* Direct3DCreate9Func)(UINT SDKVersion);
        Direct3DCreate9Func d3d9Create = (Direct3DCreate9Func)GetProcAddress(d3d9Module, "Direct3DCreate9");

        if (d3d9Create)
        {
            IDirect3D9* d3d9 = d3d9Create(D3D_SDK_VERSION);
            if (d3d9)
            {
                D3DPRESENT_PARAMETERS d3dpp = {};
                d3dpp.Windowed = TRUE;
                d3dpp.SwapEffect = D3DSWAPEFFECT_DISCARD;
                d3dpp.hDeviceWindow = GetDesktopWindow();

                IDirect3DDevice9* device = nullptr;
                if (SUCCEEDED(d3d9->CreateDevice(
                    D3DADAPTER_DEFAULT,
                    D3DDEVTYPE_HAL,
                    GetDesktopWindow(),
                    D3DCREATE_HARDWARE_VERTEXPROCESSING,
                    &d3dpp,
                    &device)))
                {
                    void* deviceVTable = *(void**)device;
                    HookVirtualMethod(
                        (PVOID*)&originalD3D9Present,
                        (PVOID)HookedD3D9Present,
                        deviceVTable,
                        17
                    );

                    device->Release();
                }
                d3d9->Release();
            }
        }
    }

    // ...
}

I have tested collecting the address from a different process and my hooks worked correctly:

void HookVirtualMethod(PVOID* ppOriginal, PVOID pDetour) 
{
    DetourTransactionBegin();
    DetourUpdateThread(GetCurrentThread());
    DetourAttach(ppOriginal, pDetour);
    r = DetourTransactionCommit();
}
// ...

    originalD3D9Present      = reinterpret_cast<D3D9Present>(0x00007ffd891c8c90);
    originalD3D11Draw        = reinterpret_cast<D3D11Draw>(0x00007ffdbd7cd0b0);
    originalD3D11DrawIndexed = reinterpret_cast<D3D11DrawIndexed>(0x00007ffdbd7cdc30);
    originalDXGIPresent      = reinterpret_cast<DXGIPresent>(0x00007ffdbe2e18c0);
    
    
    HookVirtualMethod((PVOID*)&originalD3D9Present, (PVOID)HookedD3D9Present);
    HookVirtualMethod((PVOID*)&originalD3D11Draw, (PVOID)HookedD3D11Draw);
    HookVirtualMethod((PVOID*)&originalD3D11DrawIndexed, (PVOID)HookedD3D11DrawIndexed);
    HookVirtualMethod((PVOID*)&originalDXGIPresent, (PVOID)HookedDXGIPresent);

How I could get the addresses from inside the DLL in this case?

What is your goal here? You've presented a couple your of failed attempts to solve a problem, but what is the actual problem? — David Heffernan
– David Heffernan, Commented Jan 21 at 7:57
Create thread and call in it InitializeHooks() but not wait for it inside dll entry point — RbMm
– RbMm, Commented Jan 21 at 8:11
DllMain is not a thread, it is an entry point. And thread creation is explicitly listed among things you should never perform from within DllMain. Whatever you are trying to do here, trying to do it within DllMain is certainly not a good idea. — user7860670
– user7860670, Commented Jan 21 at 9:54
@user7860670 thread creation is explicitly listed among things you should never perform from within DllMain. - this is absolute nonsense. of course we can create thread from dll entry point — RbMm
– RbMm, Commented Jan 21 at 12:53
@RbMm "this is absolute nonsense" - this is official documentation. — user7860670
– user7860670, Commented Jan 21 at 13:18

marc_s · Accepted Answer · 2025-04-06 14:20:32Z

1

When a DLL is loaded, between the call to LoadLibrary(Ex) and its return, there are some steps which the Operating System does. One of those is to load the further dependencies of other Dlls, and to initialize the Dynamic data for the DLL which needs code to execute.

During this process the number of Loaded Dlls is sort of indeterminate - if my DLL tried to load a Dll, which was being loaded as an already calculated dependency then, it may load the Dll twice or not at all.

The Windows system has the LoaderLock this lock ensures only one thread is able to modify the loaded Dlls, and is owned at the point when LoadLibrary starts its work.

This means that other threads which Load libraries, will be unable to until the LoadLibrary returns, and that is the cause of your deadlock when you are using Detours.

GetProcAddress, GetModuleHandle and various other queries on Dll state will also be blocked on the LoaderLock

The other thread can't progress, because DllMain is active, and the LoaderLock held. DllMain thread can't progress because it is waiting for the thread to complete its work.

The C++ standard brushes very loosely over the behaviour of LoadLibrary and dlopen. At the end of the calls, the code is loaded, but during the calls, makes no guarantees of the environment the code runs in.

Global data in a Dll is also initialized (if it needs a function) during this period, so a global variable will not help you.

A function static is created when the function is called, so it may be possible to use that as a deferral to when the Dll is loaded.

edited Apr 6 at 14:20

marc_s

760k186 gold badges1.4k silver badges1.5k bronze badges

answered Jan 26 at 10:54

mksteve

13.4k3 gold badges31 silver badges53 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

MSalters Apr 7 at 9:30

"The C++ standard brushes very loosely over the behaviour of LoadLibrary and dlopen". That's still a pretty strong statement. As far as the ISO C++ Standard is concerned, LoadLibrary is an OS function, and an OS function could format your hard drive.

MSalters Apr 7 at 9:33

"If my DLL tried to load a Dll, which was being loaded as an already calculated dependency then, it may load the Dll twice or not at all.". To clarify: this is the case when you try to call LoadLibrary in DllMain under loader lock. Besides loading incorrectly, other possible outcomes are crashing or deadlocking. If your DLL needs another DLL, it needs to depend explicitly on that DLL, and Windows will correctly serialize the loading.

Collectives™ on Stack Overflow

Getting function addresses from a DllMain hook

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related