2023年6月22日发(作者：)

【渲染逆向】HookD3DAPI防⽌被跨省，只放⼀个预览前⾔  RenderDoc等⼀系列抓帧⼯具的原理，是在运⾏前，在图形API初始化之前将⾃⼰的dll注⼊到⽬标程序中，并hook⼀系列图形API，得到API调⽤时的参数。如果在图形API初始化之后Hook，很可能出现⽆法检测到图形API（打开UI后，上⾯的API检测显⽰none）。  我们⾃⼰写的图形程序如果没加保护，能直接⽤RenderDoc注⼊进去。注⼊⽅式  不管是注⼊⾃⼰写的dll，还是renderdoc等已经写好的dll，都需要⼀个注⼊器。现有的注⼊器，例如RemoteDLL就很不错，简单轻便：RemoteDLL  打开程序后，选择⽬标进程，然后选择DLL，点击注⼊。  注意：注⼊⽬标程序、注⼊器、DLL，三者的32位、64位必须⼀致，并且如果⽬标程序有管理员权限，注⼊器没有，就可能导致注⼊器找不到⽬标程序。  这只是基本需求，但图形API的特殊性，很多函数需要在图形API初始化之前Hook，这就需要在程序打开的第⼀时间注⼊dll，因此我们需要⾃⼰实现⼀个注⼊器。  实现基本注⼊器需要引⼊的头⽂件：#include #include #include     ⾸先我们设定要打开程序的exe路径、命令⾏参数、以及⼯作⽬录，后两者并不是必须的。命令⾏参数⾃然不必多说，⼯作⽬录⼀般是我们⾃⼰写Visual Studio时，编译链接出来的exe才会和⼯作路径不⼀致，后两者如果没有特殊需求，都可以是NULL。TCHAR szExePath[] = TEXT("你的exe路径");//你的EXE路径TCHAR szForceDX12Cmdline[] = TEXT("-force-d3d12");//你的命令⾏参数，这个事例参数是强制unity游戏以d3d12运⾏，可以根据需要更改TCHAR szWorkspace[] = TEXT("程序的⼯作⽬录");//程序的⼯作⽬录  然后利⽤CreateProcess创建进程，打开程序：//CreateProcess的返回值BOOL bSuccess = FALSE;//CreateProcess传出的进程信息PROCESS_INFORMATION pi;STARTUPINFO si;ZeroMemory(&pi, sizeof(PROCESS_INFORMATION));ZeroMemory(&si, sizeof(STARTUPINFO)); = sizeof(STARTUPINFO);s |= STARTF_USESTDHANDLES;bSuccess = CreateProcess( szExePath,//exe路径 szForceDX12Cmdline,//命令⾏参数 NULL, NULL, TRUE, 0, NULL, szWorkspace,//⼯作路径 &si, &pi);if (!bSuccess){ std::cout << "创建失败" << std::endl;}else{ std::cout << "成功，进程号为：" << essId << std::endl;}  这样就能打开进程（如果成功），并通过PROCESS_INFORMATION对象得到进程信息。  然后我们写⼀个注⼊⽅法，参数是dll的地址和⽬标进程号:BOOL Inject(LPCTSTR DLLPath, DWORD ProcessID)。  我们选择远程线程注⼊⽅法。每个进程之间的空间彼此隔离，注⼊器⽆法操控⽬标进程的空间，但可以通过开启⼀个远程线程的⽅法。  我对远程线程注⼊并不能描述的很明⽩，但⽹上资料有很多，这⾥放上我的代码：BOOL Inject(LPCTSTR DLLPath, DWORD ProcessID){ HANDLE hProcess = nullptr; hProcess = OpenProcess(PROCESS_ALL_ACCESS, TRUE, ProcessID); if (!hProcess) { std::cout << "打开⽬标进程句柄失败" << std::endl; return FALSE; } SIZE_T PathSize = (_tcslen(DLLPath) + 1) * sizeof(TCHAR); LPVOID StartAddress = VirtualAllocEx(hProcess, NULL, PathSize, MEM_RESERVE | MEM_COMMIT, PAGE_EXECUTE_READWRITE); if (!StartAddress) { std::cout << "申请路径地址空间失败" << GetLastError() << std::endl; return FALSE; } if (!WriteProcessMemory(hProcess, StartAddress, DLLPath, PathSize, NULL)) { std::cout << "传⼊路径地址空间失败" << std::endl; return FALSE; } PTHREAD_START_ROUTINE pfnStartAddress = (PTHREAD_START_ROUTINE)GetProcAddress(GetModuleHandle(_T("")), "LoadLibraryW"); if (!pfnStartAddress) { std::cout << "获取LoadLibraryW函数地址失败" << std::endl; return FALSE; } HANDLE hThread = CreateRemoteThreadEx(hProcess, NULL, NULL, pfnStartAddress, StartAddress, NULL, NULL, NULL); if (!hThread) { std::cout << "打开远程线程失败" << std::endl; return FALSE; } WaitForSingleObject(hThread, INFINITE); CloseHandle(hThread); CloseHandle(hProcess); return TRUE;}  ⼤概是找到的LoadLibraryW⽅法的地址，并调⽤加载DLL。  最后注⼊就完事了：TCHAR RenderDocDll[] = TEXT("我的的地址");if (!Inject(RenderDocDll, essId)) std::cout << "创建远程线程失败" << std::endl;else std::cout << "成功创建远程线程" << std::endl;CloseHandle(ss);CloseHandle(d);  这样就能通过远程注⼊的⽅式，将renderdoc注⼊图形API程序中，分析渲染过程。  但假如有⼈不⽌想做这么多呢？  在在SwapChain执⾏Present前，对RenderTarget进⾏⼀次后处理，岂不是能做出类似调⾊个功能？  如果把⼈物渲染DrawCall的深度测试关闭，岂不是就能透视？  很多外挂或图形调整插件都是这么做的。我们也可以⾃⼰实现⼀个dll，⽤来hook图形API。inline hook DLL  ⾸先要引⼊⼀票头⽂件：#include //提供_beginthreadex函数#include //⽤于拍摄快照，检查当前进程已经加载了哪些dll#include #include //d3d的头⽂件#include #include #include //STL#include #include #include #include   d3dx11tex.h是我⽤来保存RT的，需要在微软下载DirectX2010 SDK安装，这⾥⾯d3d相关的库都需要链接：#pragma comment(lib, "")#pragma comment(lib, "")#pragma comment(lib, "")  如果找不到链接lib⽂件，还要到项⽬属性>链接器>常规>附加库⽬录中，把lib⽬录填⼊。（不知道在哪就⽤everything找，找不到就上⽹查⼀查安装）。  除此外我也不打算⼿动实现inline hook(因为菜)，所以hook交给微软的hook库Detours就好了。下载源码后，打开VS (2017)的开发⼈员命令提⽰符（可以在VS⼯具菜单/命令⾏/开发者命令提⽰中找到），进⼊src⽬录下，输⼊nmake命令，在得到的include⽬录下得到detours.h头⽂件，并链接lib.X64⽬录下。  dll的main函数：BOOL WINAPI DllMain(HINSTANCE hInstance, DWORD fdwReason, LPVOID){ DisableThreadLibraryCalls(hInstance); switch (fdwReason) { case DLL_PROCESS_ATTACH: _beginthreadex(nullptr, 0, init, nullptr, 0, nullptr); break; } return TRUE;}  DLL_PROCESS_ATTACH是当dll注⼊进程后调⽤。  init函数⽤于初始化，我们还未实现，现在就来实现init函数。unsigned int __stdcall init(void* data){ return 0;}  嗯，这就是基础的init函数，假如在⾥⾯写⼀个MessageBox，⽤注⼊器注⼊后，就可以弹出⼀个对话框。为了⽅便我们调试，可以在⽬标进程中打开⼀个对话框：bool OpenConsole(){ if (AllocConsole()) { freopen("CONOUT$", "w", stdout); SetConsoleTitle(L"Debug Console"); SetConsoleTextAttribute(GetStdHandle(STD_OUTPUT_HANDLE), FOREGROUND_GREEN | FOREGROUND_BLUE | FOREGROUND_RED); std::cout << "Hello Inject!" << std::endl; return true; } return false;}  接下来要Hook D3D11的⽅法，有两种，⼀种是类似D3D11CreateDevice这样，作⽤域是全局的函数，另⼀种是类似IDXGISwapChain::Present这样的成员函数。先说后者。VMT Hook  我不清楚怎么找⼀个成员函数的地址，或许直接⽤类名ClassName::*MethodName，不过D3D这些成员函数都有些特殊，它们都是虚函数，地址存储在虚表中。  根据C++对象内存布局，如果对象有虚函数，那么对象最前⾯就是虚表指针vfptr，我们可以⽤x64dbg，或VS的命令⾏查看IDXGISwapChain的内存布局：解决⽅案属性>C++>命令⾏>添加 /d1 reportSingleClassLayoutIDXGISwapChain >应⽤>编译项⽬，就能看到SwapChain的布局：IDXGISwapChain内存布局  可以看到其中的18个⽅法，对应继承结构：IDXGISwapChain : public IDXGIDeviceSubObjectIDXGIDeviceSubObject : public IDXGIObjectIDXGIObject : public IUnknownIUnknown  为了⽅便查表，我们可以把⽣成的表单粘贴下来做成枚举：//D3D_VMT_Indices.h//VMT是Virtual Method Table的缩写enum class IDXGISwapChainVMT{ QueryInterface, AddRef, Release, SetPrivateData, SetPrivateDataInterface, GetPrivateData, GetParent, GetDevice, Present, GetBuffer, SetFullscreenState, GetFullscreenState, GetDesc, ResizeBuffers, ResizeTarget, GetContainingOutput, GetFrameStatistics, GetLastPresentCount};  在DX11中，除了SwapChain外，最常⽤的还有ID3D11Device和ID3D11DeviceContext的虚表⽅法，⽤同样的⽅法写出这两个类的虚表枚举。  要Hook这些虚函数，先要获取它们的虚表指针，我们获取不到⽬标程序创建的Device、Context、SwapChain对象，但好笑的是，相同类型的对象共⽤⼀个虚表指针的地址，所以我们可以创建⼀个Device、Context、SwapChain，虽然这些不能⽤于渲染，但可以得到虚表指针，然后Hook其中的虚函数，当D3D程序内部Device等对象调⽤这些函数时，会⾃⼰把⾃⼰送给我们。  当然，当前的任务还是获取虚表指针，为此我们创建这三个对象：void** g_pDeviceVMT = nullptr;void** g_pSwapchainVMT = nullptr;void** g_pDeviceContextVMT = nullptr;//⽤于创建Device、SwapChain、Context，只要能成功创建出来，参数是随意的bool GetD3D11VMT(){ //这些对象只是为了获取虚表，并不需要被使⽤ ID3D11Device* l_pDevice = nullptr; IDXGISwapChain* l_pSwapchain = nullptr; ID3D11DeviceContext* l_pDeviceContext = nullptr; DXGI_SWAP_CHAIN_DESC scd; ZeroMemory(&scd, sizeof(DXGI_SWAP_CHAIN_DESC)); Count = 1; = DXGI_FORMAT_R8G8B8A8_UNORM; = 1920; = 1080; Usage = DXGI_USAGE_RENDER_TARGET_OUTPUT; g = DXGI_MODE_SCALING_UNSPECIFIED; neOrdering = DXGI_MODE_SCANLINE_ORDER_UNSPECIFIED; = DXGI_SWAP_CHAIN_FLAG_ALLOW_MODE_SWITCH; Window = GetForegroundWindow(); tor = 60; nator = 1; fect = DXGI_SWAP_EFFECT_DISCARD; = 1; y = 0; ed = ((GetWindowLongPtr(GetForegroundWindow(), GWL_STYLE) & WS_POPUP) != 0) ? false : true; D3D_FEATURE_LEVEL featLevel; HRESULT hr = D3D11CreateDeviceAndSwapChain( nullptr, D3D_DRIVER_TYPE_REFERENCE, nullptr, 0, nullptr, 0, D3D11_SDK_VERSION, &scd, &l_pSwapchain, &l_pDevice, &featLevel, nullptr ); if (FAILED(hr)) { std::cout << "创建D3D11Device和SwapChain失败" << std::endl; return false; } l_pDevice->GetImmediateContext(&l_pDeviceContext); //获取虚表 g_pSwapchainVMT = *(void***)l_pSwapchain; g_pDeviceVMT = *(void***)l_pDevice; g_pDeviceContextVMT = *(void***)l_pDeviceContext; std::cout << "获取虚表成功" << std::endl; return true;}  此时我们就可以通过Detours Hook虚函数了，我这⾥演⽰下Present的Hook流程：//定义Present的类型，注意因为是成员虚函数，第⼀个参数要传⼊对象的地址using vfn_SwapChain_Present = HRESULT(WINAPI*) (IDXGISwapChain* pThis, UINT SyncInterval, UINT Flags);//原来Present的地址vfn_SwapChain_Present oPresent = nullptr;//替换Present的⽅法HRESULT WINAPI HookFuncSwapChainPresent(IDXGISwapChain* pThis, UINT SyncInterval, UINT Flags){ //输出⼀句话，并调⽤原来的Present⽅法 std::cout << "Hook Present" << std::endl; return oPresent(pThis, SyncInterval, Flags);}bool HookPresent(){ void** p_SwapChain_VMT = g_pSwapchainVMT; oPresent = (vfn_SwapChain_Present)(p_SwapChain_VMT[(UINT)IDXGISwapChainVMT::Present]); DetourTransactionBegin(); DetourUpdateThread(GetCurrentThread()); //主要是这⼀句，将原来的Present替换成我们的Present DetourAttach((PVOID*)&oPresent, HookFuncSwapChainPresent); DetourTransactionCommit(); return true;}  这样就能做不少事，例如我们可以Hook IDXGISwapChain::Present和ID3D11DeviceContext::DrawIndexed DrawIndexedInstanced，从⽽统计每渲染⼀帧的DrawCall数量。  我们也可以试着像RenderDoc那样，将每⼀个DrawCall后的RT保存下来。  我们可以Hook ID3D11CreateDeviceAndSwapChain直接获取Device和SwapChain并全局保存，但这个⽅法⼀会再提，我们先偷个懒，在Present第⼀次运⾏的时候，通过传递过来的SwapChain的GetDevice⽅法获取Device，然后通过Device的GetImmediateContext⽅法获取Context：//全局变量IDXGISwapChain* g_pSwapchain = nullptr;ID3D11Device* g_pDevice = nullptr;ID3D11DeviceContext* g_pContext = nullptr;bool IsInit(){ return (g_pDevice != nullptr) && (g_pSwapchain != nullptr) && (g_pContext != nullptr);}void InitD3D(IDXGISwapChain* pSwapChain){ if (!IsInit()) { g_pSwapchain = pSwapChain; pSwapChain->GetDevice(__uuidof(ID3D11Device), (void**)&g_pDevice); g_pDevice->GetImmediateContext(&g_pContext); }}//上⾯声明的 HookFuncSwapChainPresent ⽅法内加上if (!IsInit()){ InitD3D(pThis);}  然后写⼀些截屏的逻辑//全局变量bool doCapture = false;bool Capturing = false;int gCaptureNum = 0;void TriggerCapture(){ doCapture = true;}//上⾯声明的 HookFuncSwapChainPresent 中加⼊：if (Capturing)//如果上⼀帧在截屏，关闭截屏 Capturing = false;if (doCapture)//如果要截屏{ doCapture = false; Capturing = true; gCaptureNum = 0; std::cout << "截帧" << std::endl;}  要保存RT，就要获取当前RT，我们可以通过Hook Context的OMSetRenderTargets⽅法，来维护⼀个全局当前的RT变量。虽然下⾯的⽅法看起来⼀团乱⿇，但和上⾯⼀开始Hook Present的套路完全⼀样//RT数怎么也⼤不过8吧？如果⼤过也没事，反正不会频繁分配空间std::vector g_ppRenderTargetView(8);using vfn_DeviceContext_OMSetRenderTargets = void(WINAPI*)(ID3D11DeviceContext*, __in_range(0, D3D11_SIMULTANEOUS_RENDER_TARGET_COUNT) UINT NumViews, __in_ecount_opt(NumViews) ID3D11RenderTargetView* const* ppRenderTargetViews, __in_opt ID3D11DepthStencilView* pDepthStencilView);vfn_DeviceContext_OMSetRenderTargets oDeviceContext_OMSetRenderTargets = nullptr;void WINAPI HookFuncDeviceContext_OMSetRenderTargets(ID3D11DeviceContext* pThis,

__in_range(0, D3D11_SIMULTANEOUS_RENDER_TARGET_COUNT) UINT NumViews, __in_ecount_opt(NumViews) ID3D11RenderTargetView* const* ppRenderTargetViews, __in_opt ID3D11DepthStencilView* pDepthStencilView){ g_(); for (int i = 0; i < NumViews; ++i) { ID3D11RenderTargetView* pRenderTargetView = *(ppRenderTargetViews + i); if (pRenderTargetView == nullptr) continue; g__back(pRenderTargetView); } return oDeviceContext_OMSetRenderTargets(pThis, NumViews, ppRenderTargetViews, pDepthStencilView);}bool HookDeviceContext_OMSetRenderTargets(){ void** p_DeviceContext_VMT = g_pDeviceContextVMT; oDeviceContext_OMSetRenderTargets = (vfn_DeviceContext_OMSetRenderTargets)p_DeviceContext_VMT[(UINT)ID3D11DeviceContextVMT::OMSetRenderTargets]; DetourTransactionBegin(); DetourUpdateThread(GetCurrentThread()); DetourAttach((PVOID*)&oDeviceContext_OMSetRenderTargets, HookFuncDeviceContext_OMSetRenderTargets); DetourTransactionCommit(); return true;}  有了RT就能得到资源（Resource），就有办法保存：void CaptureFrame(){ if (g_() == 0) return; ++gCaptureNum; for (int i = 0; i < g_(); ++i) { std::wstringstream wss; wss << L"我的保存路径Image_" << gCaptureNum << "_RT" << i << ".dds";

ID3D11RenderTargetView* view = g_ppRenderTargetView[i]; if (view == nullptr) continue; ID3D11Resource* pSourceResource; view->GetResource(&pSourceResource); D3D11_RENDER_TARGET_VIEW_DESC rtvDesc; view->GetDesc(&rtvDesc); std::cout << "Resource DXGIFormat: " << magic_enum::enum_name() << std::endl; HRESULT hr = D3DX11SaveTextureToFile(g_pContext, pSourceResource, D3DX11_IFF_DDS, ().c_str()); if (SUCCEEDED(hr)) std::wcout << "Save To " << () << std::endl; else std::cout << "截图错误：" << hr << std::endl; pSourceResource->Release(); }}  我试过在某个新游戏（对现在来说）⽤BMP格式保存，可惜只有渲染UI时能正常保存，PNG也是，不过DDS格式竟然能正常保存。  然后是体⼒活，要Hook Context的DrawIndexed和DrawIndexed，如果有必要，还有Draw和DrawInstanced，这些⽅法中先调⽤DrawCall，然后调⽤CaptureFrame，这⾥我放下Hook DrawIndexed的事例：using vfn_DeviceContext_DrawIndexed = void(STDMETHODCALLTYPE*)(ID3D11DeviceContext*, UINT, UINT, UINT);vfn_DeviceContext_DrawIndexed oDrawIndexed = nullptr;void STDMETHODCALLTYPE HookFuncDeviceContextDrawIndexed(ID3D11DeviceContext* Context, UINT IndexCount, UINT StartIndexLocation, UINT BaseVertexLocation){ oDrawIndexed(Context, IndexCount, StartIndexLocation, BaseVertexLocation); if (Capturing) CaptureFrame();}bool HookDrawIndexed(){ void** p_DeviceContext_VMT = g_pDeviceContextVMT; oDrawIndexed = (vfn_DeviceContext_DrawIndexed)(p_DeviceContext_VMT[(UINT)ID3D11DeviceContextVMT::DrawIndexed]); DetourTransactionBegin(); DetourUpdateThread(GetCurrentThread()); DetourAttach((PVOID*)&oDrawIndexed, HookFuncDeviceContextDrawIndexed); DetourTransactionCommit(); return true;}全局空间函数 Hook  和上⾯基本同样的套路，我Hook CreateWindowExW：using fn_CreateWindowExW = HWND(WINAPI*)( DWORD, LPCWSTR, LPCWSTR, DWORD, int, int, int, int, HWND, HMENU, HINSTANCE, LPVOID );fn_CreateWindowExW oCreateWindowExW = CreateWindowExW;HWND WINAPI HookFuncCreateWindowExW(DWORD dwExStyle, LPCWSTR lpClassName, LPCWSTR lpWindowName, DWORD dwStyle, int X, int Y, int nWidth, int nHeight, HWND hWndParent, HMENU hMenu, HINSTANCE hInstance, LPVOID lpParam){ std::wcout << L"HookCreateWindowExW! WindowName: " << lpWindowName << std::endl; std::cout << "X: " << X << ", Y:" << Y << ", width: " << nWidth << ", height: " << nHeight << std::endl; auto res = oCreateWindowExW( dwExStyle, lpClassName, lpWindowName, dwStyle, X, Y, nWidth, nHeight, hWndParent, hMenu, hInstance, lpParam); return res;}bool HookCreateWindowExW(){ DetourTransactionBegin(); DetourUpdateThread(GetCurrentThread()); DetourAttach((PVOID*)&oCreateWindowExW, HookFuncCreateWindowExW); DetourTransactionCommit(); return true;}  嗯，这是能运作的，可惜我Hook ID3D11CreateDevice和ID3D11CreateDeviceAndSwapChain时获取不到，或许是做了Hook保护？还是因为CreateWindowExW是，地址空间所有进程⼀样？  我不清楚，但⽤了另⼀种⽅法成功了：oD3D11CreateDevice =

(fn_D3D11CreateDevice)GetProcAddress(GetModuleHandle(_T("")), "D3D11CreateDevice");  我可以⽤这个开启Debug Layer：HRESULT WINAPI HookFuncD3D11CreateDevice( _In_opt_ IDXGIAdapter* pAdapter, D3D_DRIVER_TYPE DriverType, HMODULE Software, UINT Flags, _In_reads_opt_(FeatureLevels) CONST D3D_FEATURE_LEVEL* pFeatureLevels, UINT FeatureLevels, UINT SDKVersion, _COM_Outptr_opt_ ID3D11Device** ppDevice, _Out_opt_ D3D_FEATURE_LEVEL* pFeatureLevel, _COM_Outptr_opt_ ID3D11DeviceContext** ppImmediateContext){ std::cout << "Hook D3D11CreateDevice!" << "Flag: " << Flags << std::endl; if (Flags == 1) Flags |= D3D11_CREATE_DEVICE_DEBUG; HRESULT hr = oD3D11CreateDevice(pAdapter, DriverType, Software, Flags, pFeatureLevels, FeatureLevels, SDKVersion, ppDevice, pFeatureLevel, ppImmediateContext ); return hr;}总结  通过这样的⽅法，我hook到游戏中并截帧，不过这个⽅法有很多缺陷，例如那个游戏在渲染数多时，有1500左右DrawCall，因为⽤了延迟管线，不少DrawCall都是MRT，最终保存的RT数要乘上3、4倍的DrawCall数，14G左右，尽管是三星SSD硬盘，也运⾏了3-5分钟，保存下来的DDS图⽚也未必是都能看的，VisualStudio和RenderDoc各能读取⼀些。  注⼊器的编写也碰到过⼀些问题，通过把注⼊器改成系统⽂件名解决了……  如果有办法，我还是想通过注⼊RenderDoc的⽅式分析并截帧，可惜现在注⼊后，能在dll列表中看到，但并没有起作⽤，或许我要去阅读⼀下RenderDoc的源码。