Video display shader support

VirtualDub has a special video display driver called the D3D FX driver that allows the power of a 3D accelerator to be unleashed when displaying video. This is only for display purposes — the result can't be saved to disk — but it is useful for improving display quality as well as experimenting with different image processing algorithms. The power and ease of programming for modern GPUs makes it possible to prototype a shader for an algorithm in minutes that runs in real-time.

Requirements and enabling the driver

A 3D accelerator with hardware support for vertex and pixel shaders is required to use the Direct3D FX display driver in VirtualDub. A video card which supports at least pixel shader 2.0 is highly recommended, as pixel shader models 1.1-1.4 are very restricted in the amount of computation and texture fetches that can be performed in a single pass.

In addition, you must have d3dx9_25.dll, the D3DX DLL from the DirectX 9.0c April 2005 release, installed. This DLL does not come with VirtualDub and must be installed from the DirectX redistributable; currently it is not installed by the standard DirectX 9.0c install on Windows Update. As of July 3, 2006, the April 2005 D3DX dll redistributable is available as directx_9c_Apr05sdk_redist.exe at the following URL:

http://www.microsoft.com/downloads/details.aspx?FamilyId=402111C4-6478-46C1-BBD9-1BF9FCCE52F4&displaylang=en

Note that d3dx9_25.dll is a system DLL — it is intended to be installed only by the DirectX Setup installer into the Windows\System32 directory.

To enable the FX driver, go to Options > Preferences > Display. DirectX support, Direct3D support, and the FX driver should all be enabled. The filename of an .fx file must be supplied to use; if a full path is not given, the file is assumed to come from VirtualDub's program directory.

For detailed documentation on the .fx file format, consult the Microsoft DirectX 9.0c SDK.

Available surfaces and textures

A total of three textures and one surface are available for use:

The output render target, sized to the output frame.
The source texture, which holds the current video frame.
The previous source texture, which holds the last video frame.
Two temporary render target textures, which are at least as large as the desktop.

Declaring variables with specific names will automatically cause them to be bound to the textures:

texture vd_srctexture;
texture vd_prevsrctexture;   // new - 1.7.1
texture vd_prevsrc2texture;  // new - 1.7.3
texture vd_temptexture;
texture vd_temptexture2;

The output render target is at least X8R8G8B8. The temporary render targets are of format A8R8G8B8; in addition, they are guaranteed to be powers of two as long as the device does not support full non-power-of-two textures; in particular, the restrictions of the NONPOW2CONDITIONAL caps bit do not have to be followed when using them.

The vd_srctexture variable can take a single annotation:

bool vd_forceframeupload = force_flag; // new - 1.7.1: If true, this annotation changes the behavior of image upload when operating in field mode. Normally, when field display mode is enabled, the even and odd fields of the source texture are alternately updated; this effectively applies weave deinterlacing to the input. When vd_forceframeupload is set to true, both fields are uploaded on the first field of every frame.

Technique selection and execution

A technique must be named either point, bilinear, or bicubic for it to be used. Each of these names maps to one of the filtering modes in the right-click context menu of the video pane; this allows up to three techniques to be selected from the .fx file. If a technique is not available, a nearby available technique from the three is used instead.

When a video frame is displayed, VirtualDub sequentially executes each of the passes in the file. If interlaced display mode is enabled, the technique is executed twice per frame, after each field is updated. This allows field-savvy shaders to do adaptive deinterlacing on the video input.

Each pass is executed with a quad (four vertex rectangular mesh). The components of the vertex declaration accessible from the vertex shader are (all are two component):

POSITION: The four corners of the output viewport. These are already corrected for the Direct3D half-pixel offset so that they exactly encompass the screen.
TEXCOORD0: The four corners of the source image subrect within the source texture.
TEXCOORD1: Full-texture analogs of the first texcoord set — (0,0), (0,1), (1,0), and (1,1).

Thus, passing POSITION and TEXCOORD0 through is enough to do a straight blit using point or bilinear sampling.

Pass annotations

Passes within a rendering technique can be annotated to instruct VirtualDub to take certain actions prior to executing that pass.

float4 vd_clear = { red, green, blue, alpha };

Clears the render target to the given color. All channel values are normalized and range from 0-1.

string vd_target = "temp"; string vd_target = "temp2";

Selects the render target texture to use for rendering. If this is not specified, the output render target is used instead. The viewport is automatically set to the entire texture.

int vd_fieldmask = mask; // new - 1.7.1

Restricts a pass to running in only certain fields during field playback:

1 - run during even field only
2 - run during odd field only
3 - run during either field

If this annotation is absent, the pass always runs.

This is an example of a pass with annotations:

pass p0
<
	string vd_target = "temp";
	float4 vd_clear = { 0, 0, 0, 0 };
>
{
	VertexShader = compile vs_2_0 VS();
	PixelShader = compile ps_2_0 PS();
}

Exported variables

Declaring global variables with specific names will cause those variables to automatically be filled in by VirtualDub with data useful for rendering. These values are constant throughout the technique execution, as the source values do not change between passes.

Here are the supported variables:

float4 vd_vpsize;

Viewport size and inverse viewport size. This is the size of the output window, in pixels. vd_vpsize.xy gives the width and height of the viewport; vd_vpsize.wz gives 1/width and 1/height.

float4 vd_texsize;

Texture size and inverse texture size for the source texture. This is the size of the input source image texture, in texels (it is not the source image size). vd_texsize.xy gives the width and height of the texture; vd_texsize.wz gives 1/width and 1/height. This is useful for computing UV coordinates.

float4 vd_srcsize;

Size and inverse size of the source image. This is the size of the video frame, which is in the top-left subrect of the source texture. vd_srcsize.xy gives the size of the video frame, and vd_srcsize.wz gives 1/width and 1/height.

float4 vd_tempsize; float4 vd_temp2size;

Texture size and inverse texture size for the primary and secondary render target textures, respectively.

float4 vd_vpcorrect;

Screen space mapping and correction for the viewport. pos * vd_vpcorrect.xy + vd_vpcorrect.wz transforms pos from screen space coordinates, where (0,0) is the bottom-left corner and vd_vpsize.xy is the top-right corner, to the correct normalized display coordinates (NDC) for the output render target.

The annoying half-pixel shift imposed by Direct3D is taken care of in the translation — the screen space coordinates established by this transform use OpenGL-style pixel placement, with pixel centers at half-integers.

float4 vd_vpcorrect2;

This is the same as vd_vpcorrect, except that the coordinate mapping is inverted so that (0,0) is the top-left.

float4 vd_tvpcorrect; float4 vd_tvpcorrect2; float4 vd_t2vpcorrect; float4 vd_t2vpcorrect2;

These are the analogous screen space mappings for the primary and secondary render target textures, respectively.

Custom textures (1.7.3+)

Annotations can be used to control the dimensions of a texture:

string ResourceType;: Specifies the Direct3D resource type of the texture. Can be 1D, 2D, 3D, or cube. The default is 2D.
float2/3 Dimensions;: Specifies the width, height, and depth of a texture, in texels. The texture size is automatically adjusted to powers of two as needed.
string Format;: Specifies the Direct3D format of a texture. The format name is the same as the D3DFORMAT constant name, without the D3DFMT_ prefix, i.e. A8R8G8B8.
string Function;: Specifies the name of a function to use as a texture shader for initializing the texture.
int MIPLevels;: Specifies the number of mip map levels to allocate. A value of zero means to allocate a full mip chain.
float2 ViewportRatio;: Specifies a pair of ratios so that the size of the texture is proportional to the viewport size. If a texture shader is specified, it is re-evaluated every time the texture size is updated.
int width;: Specifies the width of the texture, in texels. This is used only if the Dimensions annotation is not present.
int height;: Specifies the height of the texture, in texels. This is used only if the Dimensions annotation is not present.

The type of a texture can also be specified by semantic:

RenderColorTarget: Specifies that the texture should be allocated with Direct3D RENDERTARGET usage.
RenderDepthStencilTarget: Specifies that the texture should be allocated with Direct3D DEPTHSTENCIL usage.

Whenever a custom texture is created, a mirror variable with the suffix _size with type float or vector is checked for. If this variable is present, its x and y components are filled with the width and height in texels, and w and z with the reciprocal width and height (note that the order is w and z, not z and w).

Texture shaders

D3DX texture shaders can be used to precalculate a texture prior to video display. These shaders are only run once when the .fx file is loaded, and the result then reused for each video frame. As they run on the CPU, they can use shader features not necessarily supported by the GPU, and are thus useful for computing lookup tables to work around the feeble calculation abilities of lower shader models.

VirtualDub uses a set of annotations on the texture object to control texture shader execution. These annotations are compatible with those used by the EffectEdit tool. The generated texture is always of the format A8R8G8B8.

float width; float height;: Sets the width and height of the texture. These are not adjusted to device requirements, so it is safest to use powers of two.
string target;: The profile to use when compiling the texture shader. It defaults to tx_1_0 if absent.
string function;: The name of the HLSL function to use for the texture shader.

Here is an example of a texture shader in use:

texture proceduraltex
<
	string function = "gen";
	int width = 16;
	int height = 256;
>;

float4 gen(float2 texcoord : POSITION,
           float2 texelSize : PSIZE) : color0
{
	return texcoord.xyxy;
}

Starting with 1.7.3, texture shaders can be applied to volume and cube textures.

Caveats

VirtualDub does not (cannot) do emulation of pixel shader models that are not supported by the GPU. This is particularly important for older video cards only supporting pixel shader 1.1-1.4, which are much more restrictive than pixel shader 2.0 or 3.0. Some restrictions to watch out for:

ps_1_x does not have a way to normalize a vector or retrieve its length.
ps_1_1 through ps_1_3 can't do dependant texture reads except through the special texture addressing operations, which can only be accessed from shader assembly. A dependant texture read is a texture read based on a calculated texture coordinate, i.e. not from a texture coordinate set.
ps_1_4 can do dependant texture reads, but only one layer deep, and with limited precision.

This is not to say that interesting and useful display shaders can't be written in downlevel shader models, but it requires some careful thought and coding. ps_1_4 is pretty rare as it was only introduced in the ATI RADEON 8xxx series; ps_1_1 was introduced with the NVIDIA GeForce 3 and ps_1_2/ps_1_3 with the GeForce 4, however. All cards with given shader model support can run all downlevel shaders, however, so a GeForce FX 5800 can run ps_1_4 shaders even though its highest supported model is ps_2_a.

An example shader

This effect file produces an emboss effect on the displayed video. It should work on any video card with pixel shader support.

texture vd_srctexture;
float4 vd_texsize;

sampler src = sampler_state {
	Texture = <vd_srctexture>;
	AddressU = clamp;
	AddressV = clamp;
	MinFilter = linear;
	MagFilter = linear;
	MipFilter = linear;
};

void VS(
	float4 pos : POSITION,
	float2 uv : TEXCOORD0,
	out float4 hPos : POSITION,
	out float2 uv0 : TEXCOORD0,
	out float2 uv1 : TEXCOORD1)
{
	hPos = pos;
	uv0 = uv - vd_texsize.wz;
	uv1 = uv + vd_texsize.wz;
}

float4 PS(float2 uv0 : TEXCOORD0, float2 uv1 : TEXCOORD1) : COLOR0 {
	return tex2D(src, uv1) - (tex2D(src, uv0) - 0.5);
}

technique point {
	pass {
		VertexShader = compile vs_1_1 VS();
		PixelShader = compile ps_1_1 PS();
	}
}