The standard way to use Microsoft's WinDbg debugger is to connect two PC's with a null modem cable. This configuration is fine for the initial development of a driver. As soon as you unleash your code on the unsuspecting world, you need a slightly more sophisticated model . The documentation can be daunting because it covers every possible permutation of debugger and transport protocol. WinDbg has evolved at such breakneck speed that the documentation has struggled to keep up. The Debugger has a multitude of different users, with different needs and different backgrounds.
In this article, I will illustrate a straightforward and extensible approach to integrating an additional Remote Host computer using a network connection. I will also demonstrate how this simple model can be extended to produce debugger configurations of dazzling complexity. Using the simple steps I'll outline here you can diagnose machine specific problems from the tranquillity of your office and have multiple developers simultaneously poring over the same debugging session. All of the material presented here is covered in the WinDbg documentation, but I found it challenging extrapolating the relevant nuggets of information . I Aim to AlleViate this chore for other development.
Review of the Standard Target And Host Setup
The WinDbg documentation does a good job of describing the standard kernel debugging setup, but it is worth reviewing briefly The simplest configuration for kernel debugging uses two computers:. A Target Machine that runs the driver under test and a Host Machine that runs the debugger. Figure 1 Depicts this configuration: -
Fighe.
Target and host.
I am not going to explore this configuration in any detail. You normally place the source for your driver on the Host Machine, which is also your personal development machine. In addition, the driver symbols (a byproduct of the build process) will probably be on the Host as well. This simple configuration has the benefit of simplicity, but it severely limits our options for remoting the debug session. In an environment where multiple developers are working on a project, we need to ensure that both the Source and the Symbols Are I Central Place Rather Than on A Development Machine That Isn't Widely Accessible. Centralizing Source and Symbols Opens Up A Load of Debugging Possibilities.getting The Symbols and Source Off The Host
In this scenario we are going to clear our Source and Symbols off the Host Machine and archive all of it off onto a Server. I am going to describe how to configure the Target and Host at a genteel pace, before quickly checking that it works. ............ .י.
Fighe.
Target, Host and Server.
The Target Machine has been setup with a shrink wrap XP installation. The only "tweak" is a modified boot.ini file. The boot.ini file contains a list of the alternative operating systems that you can boot from the main partition. Right after you install Windows XP, there will be just one entry in this file. We want to modify boot.ini to enable us to hookup a debugger to the serial port. The easiest way to edit this file is to open a command prompt, change the Attributes for the file, and launch notepad:
C: />
Attrib-s -h -r boot.inic: />
Notepad boot.ini
Now Adt The Boot Settings To Add / Debugport and / Baudrate Switches in The [Operating Systems] section. [Boot Loader] Timeout = 30
DEFAULT = MULTI (0) Disk (0) RDisk (0) Partition (1) / Windows
[Operating Systems]
Multi (0) Disk (0) RDisk (0) Partition (1) / Windows = "Microsoft Windows XP Professional" / FastDetect
/ debugport = com1 / baudrate = 115200
The next time you boot the target machine, The New Switches Will Cause The Kernel To Wait Until It Can Attach To Windbg Before Proceeding.
In this configuration, you install WinDbg on the host machine. I am using version 6.1.0.0017.2, but this will undoubtedly be out of date before I reach the end of this paragraph. We need to set some environment variables to ensure that the debugger can resolve both the operating system symbols and the symbols for my drivers. We also need to ensure that the Host's baudrate matches the baud rate we specified in the Target Machine's boot.ini. COM1 is the default serial port, so we do not NEED to specify That.
Let's suppose what we've put all the symbols and source on the sysphus server. Here the environment variables we need to set up on the host machine.
_NT_DEBUG_BAUD_RATE = 115200
_NT_SYMBOL_PATH = srv * // sysphus / softshared / symbols / symstore * http://msdl.microsoft.com/download/symbols
_NT_ALT_SYMBOL_PATH = // Sysphus / SoftShared / Symbols / Killerapp / SYM
_NT_Source_Path = // sysphus / softshared / symbols / killerapp / sym
Figure 3 illustrates the Source and Symbol configuration described by these environment settings. The debugger needs to find both the symbols and the source for the binaries running on the Target. We do not particularly care how they got onto the server. They could have been transferred to the server from a build on the Host Machine, or they may have been archived here. If the debugger running on the Host needs to match up OS symbols it will first look in Symstore on the Sysphus server. If this search fails, the debugger will attempt to transfer the symbols from the Microsoft symbol server Storing OS symbols on a local server means that other developers will not have to download symbols from the Microsoft server -. they can point their debugger at the server and retrieve the local copies. This Saves Both Time and Disk Space.figure 3.
Symbols and source.
Let's go!
Let's just test that this actually works. I have my latest drivers installed on the Target Machine and WinDbg is waiting to be invoked on the Host. The hardware is waiting to be plugged in and everything's ready to roll. The target is turned off.
Invoke Windbg on The Host. Enter this Command from A Command Prompt:
C: />
WINDBG -K
Windbg Should Start Up and Display
Microsoft (R) Windows Debugger Version 6.1.0017.2
CopyRight (c) Microsoft Corporation. All Rights Reserved.
Opened //./com1
Waiting to reconnect ...
Turn on the target. Windbg Will Display The Following
Connected to Windows XP 2600 x86 Compatible Target, PTR64 FALSE
Kernel Debugger Connection Establish.
Symbol search path is: //sysphus/softshared/symbols/killerapp/sym;srv*/sysphus/softshared/symbols/symstore*http://msdl.microsoft.com/download/symbols
Executable search path is: Windows XP Kernel Version 2600 MP (1 procs) checked x86 compatible
Built by: 2600.xpsp1.020828-1920
Set a Break Point
BP SaintHid! Driverence
INSERT the Device
We're in business.
Fighe 4.
Contact!
The Debugger Has Correctly Grabbed The Source from The Server. Now Let's Ponder How We CAN Leverage The Server for Some Real Power Debugging.
Add a remote host
You may have wondered why I depicted the Host machine as a laptop. If you have a building full of computers, any of which is likely to have some weird interaction with your driver, using a laptop is a quick and easy way of enabling remote debugging on any machine it's attached to We need only make a slight change to the scenario we have already examined Our invocation of WinDbg is going to have to change to reflect a new role for the Host:.. we need to tell WinDbg that it has been Promoted to a server.
C: />
Windbg -server tcp: port = 3001 -k
To start the debug server we need a TCP / IP socket number. Your network administrator will be able to give you a valid number, but be prepared for lengthy discussions about firewalls and network security. Doughnuts should help. The socket number I have been given for this Host Machine is 3001. If you aRE the network administrator, or are entrusted with the grave responsibility of selecting your own socket number, see the links in the Assigning a TCP / IP socket Number Note.
Note - Assigning a TCP / IP Socket Number
THESE LINKS SHOULD HELP You Figure Out Which TCP / IP Sockets To Use:
http://www.iana.org/assignments/port-numbers
http://www.faqs.org/rfcs/rfc793.html
http://www.petri.co.co.iL/tcp_ip_links_and_articles.htm
You can still control your debug session from the Host computer. The role of the Host has been enhanced, rather than subverted, by setting it up as a server. A second developer can now join the debug session, as illustrated in Figure 5.Figure 5.
Adding a remote host.
The Second Developer Can Now Join The Debug session on the // deepthought remote host computer by Issuing this Command from a Command Prompt:
C: />
Windbg -remote TCP: Server = // jolwhqldev-xp, port = 3001 -srcpath // sysphus / softshared / Symbols / Killerapp / src
This syntax should be self explanatory, but it is worth noting that the Remote Host does not need to set up the symbols. The association between the binaries on the Target and the Symbols on the Server is set up by the "real" Host machine ( the laptop in all my pictures). in addition to describing the connection to the Host with the server name and port number, we need to ensure that our session can access the source on the server. Depending on the debugging scenario we may choose to limit The Role of the Host Machine, See The Attack of The Drones Note for more on this.
Windbg Should Respond with Something Like the Following
NT! RTLPBREAKWITHSTATUSSTRUCTION:
80AAADCC CC INT 3
Deepthought / JWR (TCP 10.1.2.4:3389) Connected At Tue Feb 25 2003
Now That The Remote Host Is Connected We can Break Into The Session and Set Break Points in EXACTLY The Same Way We Did in Our Initial Experiments.
Attack of the drones
If The Host Up as a "drone" computer. The host at all this situation. If the host is a roaming laptop, it is it is is a rotory from having access to the source. in fact we could avoid the overhead of a GUI by using KD.EXE as our Host debugger instead of WinDbg. The WinDbg Session running on the Remote Host will connect to this session in exactly the same way, regardless of whether KD or WinDbg is running on the host. We can invoke KD with exactly the same environment variable settings that we used in our initial configuration of the Host. (of course, it is not necessary to set the _NT_SOURCE_PATH variable since KD won ' t be accessing the source.) you can also append the -noio flag to ensure maximum inscrutability from KD. This will ensure that the debugger generates no output on the server machine. If you are unused to the KD debugger, this is certainly the preferred Operating mode.
The flags do differ slightly between WinDbg and KD. Since KD is never anything but a kernel debugger, we do not need to use the -k switch. However -k is used to specify the Target connection. The following examples should illustrate the point .
To launch kd on the host, using tcp port 3001 and com1:
C: />
Kd-Server TCP: Port = 3001-NOIO
To launch kd on the host, using port 3002 and com2 instead (here we need the -k switch):
C: />
Kd-Server TCP: Port = 3002 -k COM: port = com2-noio
I tend to have a machine set up like this with KD invoked in the registry run section, so that a naive or occasional user can simply turn on the machine and start a debug session, to which a developer can attach from a Remote Host.
Adding Multiple Remote HostsYou can extend the single Remote Host concept to allow Multiple Remote Hosts to join the debugging session. The scenario I presented above is geared toward the lone developer. Additional developers can join the same debug session in exactly the same way. Figure 6 Shows An Additional Remote Host On The Network.
Fighe 6.
Add aNother Remote Host.
WE CAN Connect To The Real Host Is Exactly The Same As We Did in "Adding a remote host". You can all the window..
Using the windbg gui to connect to a remote session
....................
Step 1. SELECT Connect to a Remote Session
.
Step 2. Enter the debug server and TCP / IP Socket
.
Step 3. Set up the source path
.
Add Multiple Targets to a Single Host
Adding a second Target machine is also pretty straightforward. All we need is a second com port and a second TCP / IP socket. Initially we used com1 to connect to the Target, so we will use com2 to connect up the Second Target machine. We Will Need to Invoke A Separate Windbg for Each Target. To check That this works we can start up The Host for the com1 target using the step and host setup.
Here Are The Additional Steps to Prepare for Debugging of a Second Target:
First we need to set up the new Target for local debugging on the Host. This will enable us to verify that the serial connections between the Host and new Target are correct. Recall that WinDbg defaults to COM1. Our existing Target is using COM1. The New Target Will Use Com2. so The invocation for the second instance of windbg Becomes: -c: />
Windbg -k com: port = COM2
Now we have two..............
To enable Remote Hosts to connect to either of these Debug Sessions We need a second TCP / IP socket. Recall that our original Target was assigned the socket 3001. I am going to associate 3002 with the second Target attached to com2. Shut down the new , Local Only, Debug Session and Type The Following: -
C: />
Windbg -server TCP: Port = 3002-K COM: port = COM2
Now multiple Remote Hosts can connect to either of these debug sessions by specifying the correct TCP / IP socket. This is an ideal scenario for a Test Lab. A single computer can host a debug session for each test machine, using multiple com ports.
Figure 7 shows a Basic Multiple Target Setup.
Figure 7. Multiple Targets
.
Why Would I want to do this?
Let's look at a concrete example of the multiple Target setup in action. Let's suppose I have some code that demonstrates an OS specific spin lock. (In fact, I'm using the SPINLOCK sample that accompanies Programming the Microsoft Windows Driver Model Second Edition ( Microsoft Press 2003).) If the code is running on XP, it's going to use the more efficient in-stack queued spin lock, otherwise it's going to degrade gracefully and use the old-style spin lock. I am going to prove that this Works by Stepping Through The Code Simultaneously On Windows 2000 and XP.
The Target Machine on the left in Figure 7 is running Windows XP. The machine on the right has Windows 2000 installed. I'll use ports 3001 and 3002 to handle the remote debugging sessions for these machines, respectively.Let's assume that the debuggers on The Host Have Been Set Up and Are Connected Up Correctly To Com1 And Com2. I am Going to Focus on How We Get Two Debug Sessions Running On A Remote Host. Here We go ...
Install the driver on both Target machines. In this example, you use the Add Hardware wizard for a fake device whose driver has an INF file. Turn off both Target machines. On the Remote Host, run an instance of the debugger for each Target we Want to debug. Port 3001, The XP Target Machine on COM1:
C: />
Windbg -remote TCP: Server = // jolwhqldev-xp, port = 3001 -srcpath // sysphus / softshared / Symbols / Killerapp / src
Port 3002, The Windows 2000 Target Machine on COM1:
C: />
Windbg -remote TCP: Server = // jolwhqldev-xp, port = 3002 -srcpath // sysphus / softshared / symbols / killerapp / src
Start Both Targets. During the boot sequence break Into Each Remote Host Debugger Instance and Set a Breakpoint:
Bu Spinlock! InitializespinlockFunctionPointers
Restart the debuggers.
G
Both debuggers will break when execution reaches the breakpoint we set. We can single step through the code and prove irrefutably that this code will indeed only try to use queued spin locks on Windows XP. Figure 8 is a screen shot showing two instances of WinDbg running On My Remote Host, with Execution Paused At The OS-Specific Code.
Figure 8. Which Spin Lock?
.
The Debugger Correctly Illustrate The Different Code Paths on The DiffERENT OPERATING Systems.
Summary
In this article I have a tried to steer a straight course from the simplest configuration to a point where we are ready to embark on some profoundly complex and esoteric configurations. We could employ these techniques, for example, to rig up a whole Test Lab to a single Host Computer. An army of developers could be ready and waiting to attach remotely to any Target Machine when a problem arises. The days of wandering around with a laptop and a Null Modem cable are well and truly over.As well as enabling development / test teams to work more efficiently, using WinDbg in this way is a fantastic learning tool. As I demonstrated in the last example, simultaneoulsy stepping through the same code on different operating systems is now a possibility. When you consider that any other developer with Access to the magic socket could Also Join The Debug session, The Possibilities Are Endless. WHETHER You Are A Single Developer, Whose Code Needs To Run ON A Pleta of Different Software / Hardware Configurations, Or Part of a Vast Team of Developers, these Techniques Can Be Invaluable.
About The Author:
Jolyon Wright Is Principal Software Engineer AT
Saitek PLC, Purveyors of Fine Hardware and Software Solutions for the Gaming Industry Since 1979.
