MS Visual C++ 6 Hacks
These page contains various hacks and modifications to MS Visual C++ 6. This neat IDE has outlived itself, and I still use it to this day (July 2008). Back in 2005, I made a few modifications to add a few missing features and fix some annoying ones. I finally post them here.
- Installation of SP6 on Vista x64
- Caret Extensions
- Remote Debugging Prompts
- Using Help Files for F1
Installation of SP6 on Vista x64
When installing Visual Studio Service Pack 6 on Windows Vista x64, you might encounter that the setup fails with the following message: Visual Studio 6.0 Service Pack 6 set up was not completed successfully.
To fix the problem, you need to load the following values into your system registry. Open Notepad and paste in the following:
REGEDIT4 [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\MS Setup (ACME)\Table Files] "Visual Studio 6.0 Professional Edition@v184.108.40.206.0626 (1033)"="C:\\Program Files (x86)\\Microsoft Visual Studio\\Common\\Setup\\1033\\setup.stf"
Save the file, and then double-click it to import it into your registry.
Please note that you need to change the above strings to match your edition of Visual Studio (Professional, Enterprise, etc). As well, you must ensure that the .stf file that link is pointing to is valid. This should be the .stf file that the original Visual Studio setup created.
The only caret that the editor supports is the vertical bar. Even in overwrite mode, it stays the same. Not being able to distinguish an insert caret from an overtype caret can be annoying. This patch to DEVEDIT.PKG changes the overtype caret to a solid block, as well as changes the insert caret from a vertical bar to a horizontal bar.
Here are some screenshots to illustrate the patch:
Before the modifications:
After the modifications (insert caret):
After the modifications (overtype caret):
Download the patch here (257 KB). Extract it to %ProgramFiles%\Microsoft Visual Studio 6\Common\MSDev98\Bin. Before you overwrite, make sure your original DBGEDIT.PKG is from SP6, dated 27.02.2004 01:00:00, has a size of 774,193 bytes and a CRC32 checksum of 04398C01.
How it is Implemented
We inject some code into DEVEDIT.PKG that overrides some calls to CreateCaret. From then, we check some stack variable to see if overtype mode is on, and then create either the block caret (overtype mode) or the horizontal bar caret (insert mode). I don't remember how I found that stack variable. It must have been through debugging and focus I don't have anymore.
Here are the injection points in the file. For complete listing see rawedit.asm.
devedit_50885E00: ; called by 50804775 call @F @@: pop ecx add ecx,(vaOvertype - $85E00 - $5) ; decide the appropriate caret position: ; - if we are in INS mode, the caret should appear below the character (bottom left) ; - if we are in OVR mode, the caret should appear over the character (top left) ; check if we are in OVR mode cmp byte [ecx],0 jne .no_inc ; don't increase if we are in OVR mode inc edi .no_inc: mov eax,[esi+$160] imul edi,[eax+$040] mov eax,[esi+$0C8] cmp byte [ecx],0 jne .no_inc2 ; don't increase if we are in OVR mode dec edi .no_inc2: inc ebx ; increase height retn devedit_50885E40: ; called by 508071A7, 50842F44, 5084388C call @F @@: pop ecx sub ecx,$85E40 + $5 cmp byte [ecx+vaOvertype],0 je .normal ; decide which caret to make ; - if we are in INS mode, make a horizontal line caret ; - if we are in OVR mode, make a rectangular block caret ; block caret (overwrite made) push dword [eax+40h] ; nHeight push dword [eax+54h] ; nWidth jmp .rest .normal:; regular horizontal caret (insert mode) push 2 ; nHeight push dword [eax+54h] ; nWidth .rest: ; rest of the call (static) push 0 ; hBitmap push dword [esi+20h] ; hWnd call dword [ecx+$2344];CreateCaret ($50802344) retn devedit_50885E80: ; called by 508072BA (82BA) push ebx push ebp call @F @@: pop ebp sub ebp,$85E80 + $5 + $1 + $1 ; redraw only if new settings differ from current ones mov eax,[esp+8] cmp eax,[ebp+vaOvertype] je .noredraw ; make sure we don't go into an infinite loop cmp dword [vaLock],0 ja .noredraw inc dword [vaLock] ; change caret if user pressed Insert key stdcall dword [ebp+GetAsyncKeyState],VK_INSERT and eax,1 push eax xor [ebp+vaOvertype],eax ; get handle to edit control call dword [ebp+GetFocus] mov ebx,eax ; disabling/enabling window causes caret to refresh stdcall dword [ebp+EnableWindow],ebx,0 stdcall dword [ebp+EnableWindow],ebx,1 stdcall dword [ebp+SetFocus],ebx pop eax xor [ebp+vaOvertype],eax dec dword [vaLock] .noredraw: pop ebp pop ebx mov eax,[esp+8] push dword [esp] mov [esp+4],eax retn
Remote Debugging Prompts
There is also a patch that eliminates annoying message box prompts that occur when doing remote debugging. The patch makes the prompts go to the debug output window instead.
Before the modifications:
After the modifications:
Download the patch here (214 KB). Extract it to %ProgramFiles%\Microsoft Visual Studio 6\Common\MSDev98\Bin\IDE. Before you overwrite, make sure your original DEVDBG.PKG is from SP6, dated 15.07.2000 01:00:00, has a size of 585,794 bytes and a CRC32 checksum of C4AC7930.
How it is Implemented
I fished out the offending MessageBox, found where it is being called from, and with what parameters. Then using the elite String References window in W32Dasm, I found a sample call that adds text to the output window. Then I simply converted the MessageBox call into that. The only problem here was a callout to DBGEDIT, which I needed a pointer to. I guess I got the pointer from some stack variable.
Here is the code snippet that is injected, also available in rawdbg.asm:
devdbg_50C6B900: ; called by 50C41972 push ebp call @F @@: pop ebp sub ebp,$6B900 + $5 + $1 ; call to devedit.5081B015 (takes 2 params) mov eax,[ebp+$6E610] push eax ; make space for a DWORD push esp ; address to previously pushed DWORD push eax mov ecx,[eax] call dword [ecx+44h] ; call to devedit.5081ACAD (takes 2 args) ; the pushed DWORD remains safely on stack ;mov eax,[$50C6E610] ;push esi ; buffer ;push eax ;mov ecx,[eax] ;call dword [ecx+48h] ; last call to devedit.5081AD18 (takes 6 params) pop edx ; restore the stacked DWORD mov eax,[ebp+$6E610] push 0 push 1 push 0 mov ecx,[esi] ; esi = ptr to line push ecx push edx push eax mov ecx,[eax] call dword [ecx+18h] pop ebp retn
Using Help Files for F1
The default function for the F1 key in VC6 is to start its own built-in MSDN viewer that forces you to browse the MSDN collection. Personally, I never installed the MSDN collection from Visual Studio 98. I usually just use win32.hlp from ftp://ftp.borland.com/pub/delphi/techpubs/delphi2/win32.zip (the link seems to have died a year ago or so). It is really old, has some mistakes, but covers most of the API I need to use. Plus, winhlp32.exe starts like 1000 faster than the MSDN Document Explorer.
This patch fixes %CommonProgramFiles%/Microsoft Shared/VS98/VSHELP.DLL. It simply kills the "MSDN collection is not installed" message box. Then, to start your own help file, simply add another tool to the VC6 IDE:
- Go to Tools, Customize menu.
- Go to the Tools tab page.
- Add a new tool with the following properties:
Arguments: -k $(CurText) C:\HelpFiles\win32.hlp
Change your paths accordingly, obviously.
- Note the number that your newly added tool is on that list. For ease, just drag it to the top of the list to make it number 1.
- On the same dialog, go to the Keyboard tab page.
- Under the Category dropdown, select Tools.
- Under the Commands list, select UserTool1 (or whatever the number your tool is). Assign to it the F1 key.
Download the patch here (49 KB). Extract it to %CommonProgramFiles%/Microsoft Shared/VS98/VSHELP.DLL (%CommonProgramFiles% is usually your C:\Program Files\Common Files). Overwrite VSHELP.DLL.
To double-check, make sure the VSHELP.DLL you are overwriting is dated 26.05.1998 01:00:00, has a file size of 116,938 bytes, and a CRC32 checksum of F1E77CB3.
For those interested, here is the binary diff between the original VSHELP.DLL and the patched one:
Comparing files VSHELP.BAK and VSHELP.DLL 000077FF: FF 83 00007800: 15 C4 00007801: 6C 10 00007802: 11 90 00007803: E0 90 00007804: 59 90As you can see, we are simply substituting a call instruction with a stack adjustment (add esp, 10h) padded with nops (90h).