VirtualDub tries to change settings in such a way that if the settings cause the driver to blow up, that the modified settings aren't saved — but occasionally the failure occurs later, in the form of a hang, crash, blue-screen, etc. The result is that you can't use the capture device any more, because as soon as you try entering capture mode VirtualDub auto-selects the last capture driver, and then restores the saved settings, and... well, you get the point.
Fortunately, there is an escape hatch.
Holding down a Shift key when entering capture mode will prevent VirtualDub from automatically selecting the last used capture driver. The easiest way to do this is to hold Shift when selecting Capture AVI... from the menu. Similarly, holding down Shift when selecting a capture driver will prevent any saved settings that were recorded for that driver from being restored. The errant settings can then be changed and re-saved to correct the problem.
Note that this doesn't help for any settings that the capture driver itself saves and that VirtualDub doesn't know about. In that case, you can either attempt to reinstall the driver, or find the location in the Registry where it saves those settings, and try to change them.
The "overlay" display mode on most capture devices causes the capture hardware to stream the video image directly into the display memory of the video card. Sometimes the capture device and video card don't cooperate well and the result is a lockup or blowup when overlay display mode is chosen. In most cases it is better to choose the Preview mode instead, as then VirtualDub can directly control the video display, but sometimes it is possible to switch a Video for Windows capture driver's method of overlay display to a more compatible mode.
If this is occurring, check if the capture driver has an options screen. In Windows XP, navigate as follows:
If the Settings... button is not disabled (grayed out), select it and browse through the driver's configuration dialog. Look for an option called "overlay mode" or "use DrawDib"; this will switch the driver to a slower, but more compatible method of display.
First, check that you are using the correct video input and that the capture device is configured to use the right one. On some devices the composite video input is simply the S-Video input with an adapter, so it is possible to set the video input to S-Video and still see a grayscale version of a composite video input.
If you have both S-Video as well as an "external" or "camera" input, sometimes the external/camera input works better, inexplicably.
Check the video standard in use. Attempting to capture an NTSC stream as PAL, PAL as SECAM, etc. will result in scrambled color. The video standard is normally controlled in Video settings for VFW drivers and Capture filter for WDM drivers, although the location may vary.
Check Capture > Stop Conditions and make sure no conditions have been inadvertently enabled.
Interruptions in video sources, causing a loss of video signal, can occasionally cause this problem.
On Windows NT-based platforms, locking the workstation with Ctrl+Alt+Del can cause a capture operation to stop. In general, it is best to avoid doing anything that might change desktops or the video display mode.