MS Visual C++ 6 Hacks

These page contains various hacks and modifications to MS Visual C++ 6. This neat IDE has outlived itself, and I still use it to this day (July 2008). Back in 2005, I made a few modifications to add a few missing features and fix some annoying ones. I finally post them here.

Sections

Installation of SP6 on Vista x64
Caret Extensions

How it is Implemented

Remote Debugging Prompts
1. How it is Implemented
Using Help Files for F1

Installation of SP6 on Vista x64

When installing Visual Studio Service Pack 6 on Windows Vista x64, you might encounter that the setup fails with the following message: Visual Studio 6.0 Service Pack 6 set up was not completed successfully.

To fix the problem, you need to load the following values into your system registry. Open Notepad and paste in the following:

REGEDIT4

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\MS Setup (ACME)\Table Files]
"Visual Studio 6.0 Professional Edition@v6.0.0.0.0626 (1033)"="C:\\Program Files (x86)\\Microsoft Visual Studio\\Common\\Setup\\1033\\setup.stf"

Save the file, and then double-click it to import it into your registry.
Please note that you need to change the above strings to match your edition of Visual Studio (Professional, Enterprise, etc). As well, you must ensure that the .stf file that link is pointing to is valid. This should be the .stf file that the original Visual Studio setup created.

Caret Extensions

The only caret that the editor supports is the vertical bar. Even in overwrite mode, it stays the same. Not being able to distinguish an insert caret from an overtype caret can be annoying. This patch to DEVEDIT.PKG changes the overtype caret to a solid block, as well as changes the insert caret from a vertical bar to a horizontal bar.
Here are some screenshots to illustrate the patch:

Before the modifications:

After the modifications (insert caret):

After the modifications (overtype caret):

Download the patch here (257 KB). Extract it to %ProgramFiles%\Microsoft Visual Studio 6\Common\MSDev98\Bin. Before you overwrite, make sure your original DBGEDIT.PKG is from SP6, dated 27.02.2004 01:00:00, has a size of 774,193 bytes and a CRC32 checksum of 04398C01.

How it is Implemented

We inject some code into DEVEDIT.PKG that overrides some calls to CreateCaret. From then, we check some stack variable to see if overtype mode is on, and then create either the block caret (overtype mode) or the horizontal bar caret (insert mode). I don't remember how I found that stack variable. It must have been through debugging and focus I don't have anymore.
Here are the injection points in the file. For complete listing see rawedit.asm.

devedit_50885E00:   ; called by 50804775
    call   @F
@@: pop ecx
    add    ecx,(vaOvertype - $85E00 - $5)

    ; decide the appropriate caret position:
    ; - if we are in INS mode, the caret should appear below the character (bottom left)
    ; - if we are in OVR mode, the caret should appear over the character (top left)

    ; check if we are in OVR mode
    cmp    byte [ecx],0
    jne    .no_inc    ; don't increase if we are in OVR mode
    inc    edi
.no_inc:
    mov    eax,[esi+$160]
    imul   edi,[eax+$040]
    mov    eax,[esi+$0C8]

    cmp    byte [ecx],0
    jne    .no_inc2   ; don't increase if we are in OVR mode
    dec    edi
.no_inc2:
    inc    ebx       ; increase height
    retn

devedit_50885E40:   ; called by 508071A7, 50842F44, 5084388C
    call   @F
@@: pop ecx
    sub    ecx,$85E40 + $5
    cmp    byte [ecx+vaOvertype],0
    je .normal

    ; decide which caret to make
    ; - if we are in INS mode, make a horizontal line caret
    ; - if we are in OVR mode, make a rectangular block caret

    ; block caret (overwrite made)
    push   dword [eax+40h] ; nHeight
    push   dword [eax+54h] ; nWidth
    jmp    .rest

.normal:; regular horizontal caret (insert mode)
    push   2     ; nHeight
    push   dword [eax+54h] ; nWidth

.rest:  ; rest of the call (static)
    push   0     ; hBitmap
    push   dword [esi+20h] ; hWnd
    call   dword [ecx+$2344];CreateCaret ($50802344)

    retn

devedit_50885E80:   ; called by 508072BA (82BA)
    push   ebx
    push   ebp
    call   @F
@@: pop ebp
    sub    ebp,$85E80 + $5 + $1 + $1

    ; redraw only if new settings differ from current ones
    mov    eax,[esp+8]
    cmp    eax,[ebp+vaOvertype]
    je .noredraw

    ; make sure we don't go into an infinite loop
    cmp    dword [vaLock],0
    ja .noredraw

    inc    dword [vaLock]

    ; change caret if user pressed Insert key
    stdcall dword [ebp+GetAsyncKeyState],VK_INSERT
    and    eax,1
    push   eax
    xor    [ebp+vaOvertype],eax

    ; get handle to edit control
    call   dword [ebp+GetFocus]
    mov    ebx,eax
    ; disabling/enabling window causes caret to refresh
    stdcall dword [ebp+EnableWindow],ebx,0
    stdcall dword [ebp+EnableWindow],ebx,1
    stdcall dword [ebp+SetFocus],ebx
    pop    eax
    xor    [ebp+vaOvertype],eax
    dec    dword [vaLock]

.noredraw:
    pop    ebp
    pop    ebx
    mov    eax,[esp+8]
    push   dword [esp]
    mov    [esp+4],eax
    retn

Remote Debugging Prompts

There is also a patch that eliminates annoying message box prompts that occur when doing remote debugging. The patch makes the prompts go to the debug output window instead.

Before the modifications:

After the modifications:

Download the patch here (214 KB). Extract it to %ProgramFiles%\Microsoft Visual Studio 6\Common\MSDev98\Bin\IDE. Before you overwrite, make sure your original DEVDBG.PKG is from SP6, dated 15.07.2000 01:00:00, has a size of 585,794 bytes and a CRC32 checksum of C4AC7930.

How it is Implemented

I fished out the offending MessageBox, found where it is being called from, and with what parameters. Then using the elite String References window in W32Dasm, I found a sample call that adds text to the output window. Then I simply converted the MessageBox call into that. The only problem here was a callout to DBGEDIT, which I needed a pointer to. I guess I got the pointer from some stack variable.
Here is the code snippet that is injected, also available in rawdbg.asm:

devdbg_50C6B900:    ; called by 50C41972
    push   ebp
    call   @F
@@: pop ebp
    sub    ebp,$6B900 + $5 + $1

    ; call to devedit.5081B015 (takes 2 params)
    mov    eax,[ebp+$6E610]
    push   eax   ; make space for a DWORD
    push   esp   ; address to previously pushed DWORD
    push   eax
    mov    ecx,[eax]
    call   dword [ecx+44h]
    ; call to devedit.5081ACAD (takes 2 args)
    ; the pushed DWORD remains safely on stack
    ;mov     eax,[$50C6E610]
    ;push    esi     ; buffer
    ;push    eax
    ;mov     ecx,[eax]
    ;call    dword [ecx+48h]
    ; last call to devedit.5081AD18 (takes 6 params)
    pop    edx    ; restore the stacked DWORD
    mov    eax,[ebp+$6E610]
    push   0
    push   1
    push   0
    mov    ecx,[esi]  ; esi = ptr to line
    push   ecx
    push   edx
    push   eax
    mov    ecx,[eax]
    call   dword [ecx+18h]

    pop    ebp
    retn

Using Help Files for F1

The default function for the F1 key in VC6 is to start its own built-in MSDN viewer that forces you to browse the MSDN collection. Personally, I never installed the MSDN collection from Visual Studio 98. I usually just use win32.hlp from ftp://ftp.borland.com/pub/delphi/techpubs/delphi2/win32.zip (the link seems to have died a year ago or so). It is really old, has some mistakes, but covers most of the API I need to use. Plus, winhlp32.exe starts like 1000 faster than the MSDN Document Explorer.
This patch fixes %CommonProgramFiles%/Microsoft Shared/VS98/VSHELP.DLL. It simply kills the "MSDN collection is not installed" message box. Then, to start your own help file, simply add another tool to the VC6 IDE:

Go to Tools, Customize menu.
Go to the Tools tab page.
Add a new tool with the following properties:
Command: C:\WINDOWS\winhlp32.exe
Arguments: -k $(CurText) C:\HelpFiles\win32.hlp

Change your paths accordingly, obviously.
Note the number that your newly added tool is on that list. For ease, just drag it to the top of the list to make it number 1.
On the same dialog, go to the Keyboard tab page.
Under the Category dropdown, select Tools.
Under the Commands list, select UserTool1 (or whatever the number your tool is). Assign to it the F1 key.

Download the patch here (49 KB). Extract it to %CommonProgramFiles%/Microsoft Shared/VS98/VSHELP.DLL (%CommonProgramFiles% is usually your C:\Program Files\Common Files). Overwrite VSHELP.DLL.
To double-check, make sure the VSHELP.DLL you are overwriting is dated 26.05.1998 01:00:00, has a file size of 116,938 bytes, and a CRC32 checksum of F1E77CB3.

For those interested, here is the binary diff between the original VSHELP.DLL and the patched one:

Comparing files VSHELP.BAK and VSHELP.DLL
000077FF: FF 83
00007800: 15 C4
00007801: 6C 10
00007802: 11 90
00007803: E0 90
00007804: 59 90