In this article I will explain how to subclass theNotepad application window using hooks & subclassing techniques. Thistechnique can be used to build custom dll-based engines for anyapplication. In this example I will subclass the notepad applicationcreate custom menus and react when the menu is clicked. You can download ademo application which will do this
Before I start describing about cross processsubclassing ,I presume you already know what the terms hooking andwindow subclassing mean. Also I presume you have some experience workingwith hooks and subcalssing in particular and the windows SDK in general. Iwill just briefly go over these 2 techniques and explain how you couldcombine both these techniques to achieve cross-process subclassing.
This is what MSDN has to say about hooks
"In the Microsoft® Windows® operating system, a hook is a mechanism bywhich a function can intercept events (messages, mouse actions,keystrokes) before they reach an application. The function can act onevents and, in some cases, modify or discard them. Functions that receiveevents are called filter functions and are classified according tothe type of event they intercept."
You can install different types of hooks like a keyboard hook, amessage hook or a mouse hook depending upon your requirements .Hooks canbe categorized in 2 ways,
1.Global Hooks 2.Thread Specific Hooks
Global hooks are, as the name suggests Global in nature .A globalhook is installed in each and every thread that is running in thesystem. Global hooks when not properly used tend to be bulky and also slows down the system in many cases.
Thread Specific Hooks are installed in only one process. This hookcan be installed in either the same thread as the calling function (to beexplained later) or in a thread running in a different process.
Hooks are installed using the SetWindowsHookEx Api andun-installed using the UnhookWindowsHookEx Api.
Subclassing allows you to change the behavior of an existing window,typically a control, by inserting a message map to intercept the window'smessages. Subclassing is process specific, you cannot subclass awindow which is not running in the same process as your application.
Although the Windows OS does not allow us to subclass a window whichis not in the same process as our application, we can work around this byusing hooks and getting into the process space of the window that we wantto subclass.
Now let's get to building our application.
Our application will consist of 2 modules, the hooking dll whichwould install/un-install the hook and also subclass the notepadapplication and an exe which would load the hooking dll.
The Hooking Dll
The hooking dll will be used to install/uninstall windows hookand also subclass the notepad application window. Before getting into thecode for installing/uninstalling and subcalssing let's look at thefollowing piece of code
HWND hApp = NULL;
int num=0 ;// Number of the subclassed window handle ,for use in the dll
The #pragma data_seg compiler directive asks the compiler to createa data segment which can be shared by all instances of the dll.The reasonwe need this is because the dll we will be loading using our exeapplication will be in one process and the dll which would eventually hookthe notepad application will be in notepad application's process.So weneed a common data segment which can be shared by different instances ofthe dll .
The code for installing the windows hook looks something like this.
int WINAPI SetHandle(HWND HandleofTarget ,HWNDHandleofApp)
}//End this function
We would be installing a CBT hook. A CBT (Computer Based Training)hook is used when we want to be notifed when a window iscreated/destroyed/activated/minimized etc. The hook callback procedurelooks something like this
//The CBT hook Proc(Computer Based Training Hook)
LRESULT CALLBACK CBTProc(int nCode,WPARAMwParam,LPARAM lParam)
if (nCode==HCBT_ACTIVATE) //Called when the applicationwindow is activated
if((HWND)(wParam)==hTarget) //checkif the window activated is Our Target App
blnsubclassed[count]=TRUE; // Set state as subclassed
if (nCode==HCBT_DESTROYWND) //Called when the applicationwindow is destroyed
SendNotifyMessage(hApp,WM_APP +1024,(WPARAM)wParam,(LPARAM)lParam);// Sendthe message to the vb app
return CallNextHookEx(NULL, nCode, wParam, lParam);
}//End of the hook procedure
We subclass the notepad application as soon as it is activated thefirst time. The SetWindowLong Api is used to subclass the notepadapplication.The subclassed winodw procedure looks like this
//Window Procedures of the subclassed windows
LRESULT CALLBACK WindowProc(
val=count; //this gets us the exact position of this window procedure in the array
if(uMsg==273) //Message Implying Menu Clicks
result=SendNotifyMessage(hApp,WM_APP+1024,(WPARAM)(LOWORD(wParam)),(LPARAM)uMsg);// Send the message to//the hooking exe
Whenever a user clicks on a menu in notepad our hooking exe isnotified and depending on which menu was clicked our application reacts tothe click.
The Hooking Exe
The hooking exe will create the menus and load the hooking dll. The codefor doing this is as follows
MessageBox(0,"Could Not find arunning instance of Notepad.\r\nPlease Start Notepad and try again","Error",0);
AppendMenu(hAppMenu,MF_STRING + MF_POPUP,(unsigned int)hAppendMenu,"HTML");
AppendMenu(hAppendMenu,MF_STRING,126,"Add Line Break");
hLib = LoadLibrary("grimbo.dll");
hMenuWnd = GetWindow(hHookedWindow, 5);
SetHandle = (sthndl)GetProcAddress(hLib, "SetHandle");
UnSubClass = (unsub)GetProcAddress(hLib, "UnSubclass");
FillHandleArray = (filhndl)GetProcAddress(hLib, "FillHandleArray");
Thus by using hooks we can subclass a window running in any process.Cross Process Subclassing should be used very carefully , unsafehooking/subclassing generally results in crashing the applicationwhich was subcalssed/hooked.In order to test the demo, compile the dll first and then place the dll in ur windows/system32 directory and then compile the exe.