Video display shader support

VirtualDub has a special video display driver called the D3D FX driver that allows the power of a 3D accelerator to be unleashed when displaying video. This is only for display purposes — the result can't be saved to disk — but it is useful for improving display quality as well as experimenting with different image processing algorithms. The power and ease of programming for modern GPUs makes it possible to prototype a shader for an algorithm in minutes that runs in real-time.

Requirements and enabling the driver

A 3D accelerator with hardware support for vertex and pixel shaders is required to use the Direct3D FX display driver in VirtualDub. A video card which supports at least pixel shader 2.0 is highly recommended, as pixel shader models 1.1-1.4 are very restricted in the amount of computation and texture fetches that can be performed in a single pass.

In addition, you must have d3dx9_25.dll, the D3DX DLL from the DirectX 9.0c April 2005 release, installed. This DLL does not come with VirtualDub and must be installed from the DirectX redistributable; currently it is not installed by the standard DirectX 9.0c install on Windows Update. As of September 19, 2005, the April 2005 D3DX dll redistributable is available as directx_9c_Apr05sdk_redist.exe at the following URL:

http://www.microsoft.com/downloads/details.aspx?FamilyId=402111C4-6478-46C1-BBD9-1BF9FCCE52F4&displaylang=en

Note that d3dx9_25.dll is a system DLL — it is intended to be installed only by the DirectX Setup installer into the Windows\System32 directory.

To enable the FX driver, go to Options > Preferences > Display. DirectX support, Direct3D support, and the FX driver should all be enabled. The filename of an .fx file must be supplied to use; if a full path is not given, the file is assumed to come from VirtualDub's program directory.

For detailed documentation on the .fx file format, consult the Microsoft DirectX 9.0c SDK.

Available surfaces and textures

A total of three textures and one surface are available for use:

The output render target, sized to the output frame.
The source texture, which holds the video frame.
Two temporary render target textures, which are at least as large as the desktop.

Declaring variables with specific names will automatically cause them to be bound to the textures:

texture vd_srctexture;
texture vd_temptexture;
texture vd_temptexture2;

The output render target is at least X8R8G8B8. The temporary render targets are of format A8R8G8B8; in addition, they are guaranteed to be powers of two as long as the device does not support full non-power-of-two textures; in particular, the restrictions of the NONPOW2CONDITIONAL caps bit do not have to be followed when using them.

Technique selection and execution

A technique must be named either point, bilinear, or bicubic for it to be used. Each of these names maps to one of the filtering modes in the right-click context menu of the video pane; this allows up to three techniques to be selected from the .fx file. If a technique is not available, a nearby available technique from the three is used instead.

When a video frame is displayed, VirtualDub sequentially executes each of the passes in the file. If interlaced display mode is enabled, the technique is executed twice per frame, after each field is updated. This allows field-savvy shaders to do adaptive deinterlacing on the video input.

Each pass is executed with a quad (four vertex rectangular mesh). The components of the vertex declaration accessible from the vertex shader are (all are two component):

POSITION: The four corners of the output viewport. These are already corrected for the Direct3D half-pixel offset so that they exactly encompass the screen.
TEXCOORD0: The four corners of the source image subrect within the source texture.
TEXCOORD1: Full-texture analogs of the first texcoord set — (0,0), (0,1), (1,0), and (1,1).

Thus, passing POSITION and TEXCOORD0 through is enough to do a straight blit using point or bilinear sampling.

Pass annotations

Passes within a rendering technique can be annotated to instruct VirtualDub to take certain actions prior to executing that pass.

float4 vd_clear = { red, green, blue, alpha };: Clears the render target to the given color. All channel values are normalized and range from 0-1.
string vd_target = "temp"; string vd_target = "temp2";: Selects the render target texture to use for rendering. If this is not specified, the output render target is used instead. The viewport is automatically set to the entire texture.

This is an example of a pass with annotations:

pass p0
<
	string vd_target = "temp";
	float4 vd_clear = { 0, 0, 0, 0 };
>
{
	VertexShader = compile vs_2_0 VS();
	PixelShader = compile ps_2_0 PS();
}

Exported variables

Declaring global variables with specific names will cause those variables to automatically be filled in by VirtualDub with data useful for rendering. These values are constant throughout the technique execution, as the source values do not change between passes.

Here are the supported variables:

float4 vd_vpsize;

Viewport size and inverse viewport size. This is the size of the output window, in pixels. vd_vpsize.xy gives the width and height of the viewport; vd_vpsize.wz gives 1/width and 1/height.

float4 vd_texsize;

Texture size and inverse texture size for the source texture. This is the size of the input source image texture, in texels (it is not the source image size). vd_texsize.xy gives the width and height of the texture; vd_texsize.wz gives 1/width and 1/height. This is useful for computing UV coordinates.

float4 vd_srcsize;

Size and inverse size of the source image. This is the size of the video frame, which is in the top-left subrect of the source texture. vd_srcsize.xy gives the size of the video frame, and vd_srcsize.wz gives 1/width and 1/height.

float4 vd_tempsize; float4 vd_temp2size;

Texture size and inverse texture size for the primary and secondary render target textures, respectively.

float4 vd_vpcorrect;

Screen space mapping and correction for the viewport. pos * vd_vpcorrect.xy + vd_vpcorrect.wz transforms pos from screen space coordinates, where (0,0) is the bottom-left corner and vd_vpsize.xy is the top-right corner, to the correct normalized display coordinates (NDC) for the output render target.

The annoying half-pixel shift imposed by Direct3D is taken care of in the translation — the screen space coordinates established by this transform use OpenGL-style pixel placement, with pixel centers at half-integers.

float4 vd_vpcorrect2;

This is the same as vd_vpcorrect, except that the coordinate mapping is inverted so that (0,0) is the top-left.

float4 vd_tvpcorrect; float4 vd_tvpcorrect2; float4 vd_t2vpcorrect; float4 vd_t2vpcorrect2;

These are the analogous screen space mappings for the primary and secondary render target textures, respectively.

Texture shaders

D3DX texture shaders can be used to precalculate a texture prior to video display. These shaders are only run once when the .fx file is loaded, and the result then reused for each video frame. As they run on the CPU, they can use shader features not necessarily supported by the GPU, and are thus useful for computing lookup tables to work around the feeble calculation abilities of lower shader models.

VirtualDub uses a set of annotations on the texture object to control texture shader execution. These annotations are compatible with those used by the EffectEdit tool. The generated texture is always of the format A8R8G8B8.

float width; float height;: Sets the width and height of the texture. These are not adjusted to device requirements, so it is safest to use powers of two.
string target;: The profile to use when compiling the texture shader. It defaults to tx_1_0 if absent.
string function;: The name of the HLSL function to use for the texture shader.

Here is an example of a texture shader in use:

texture proceduraltex
<
	string function = "gen";
	int width = 16;
	int height = 256;
>;

float4 gen(float2 texcoord : POSITION,
           float2 texelSize : PSIZE) : color0
{
	return texcoord.xyxy;
}

Caveats

VirtualDub does not (cannot) do emulation of pixel shader models that are not supported by the GPU. This is particularly important for older video cards only supporting pixel shader 1.1-1.4, which are much more restrictive than pixel shader 2.0 or 3.0. Some restrictions to watch out for:

ps_1_x does not have a way to normalize a vector or retrieve its length.
ps_1_1 through ps_1_3 can't do dependant texture reads except through the special texture addressing operations, which can only be accessed from shader assembly. A dependant texture read is a texture read based on a calculated texture coordinate, i.e. not from a texture coordinate set.
ps_1_4 can do dependant texture reads, but only one layer deep, and with limited precision.

This is not to say that interesting and useful display shaders can't be written in downlevel shader models, but it requires some careful thought and coding. ps_1_4 is pretty rare as it was only introduced in the ATI RADEON 8xxx series; ps_1_1 was introduced with the NVIDIA GeForce 3 and ps_1_2/ps_1_3 with the GeForce 4, however. All cards with given shader model support can run all downlevel shaders, however, so a GeForce FX 5800 can run ps_1_4 shaders even though its highest supported model is ps_2_a.